Methods and system for managing and displaying virtual content in a mixed reality system

ABSTRACT

Disclosed is an approach for managing and displaying virtual content in a mixed reality environment on a one-on-one basis independently by each application, each virtual content is rendered by its respective application into a bounded volume referred herein as a “Prism.” Each Prism may have characteristics and properties that allow a universe application to manage and display the Prism in the mixed reality environment such that the universe application may manage the placement and display of the virtual content in the mixed reality environment by managing the Prism itself.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent applicationSer. No. 17/241,977, filed Apr. 27, 2021, which is a continuation ofU.S. patent application Ser. No. 16/224,719, filed Dec. 18, 2018, nowU.S. Pat. No. 11,024,086, issued on Jun. 1, 2021, which claims thebenefit under 35 U.S.C. § 119 to U.S. Provisional Patent ApplicationSer. No. 62/610,101 filed on Dec. 22, 2017, entitled “METHODS AND SYSTEMFOR MANAGING AND DISPLAYING VIRTUAL CONTENT IN A MIXED REALITY SYSTEM.”The contents of the aforementioned patent applications and patent arehereby incorporated by reference into the present application in theirentirety.

The present disclosure is related to co-owned U.S. patent applicationSer. No. 14/205,126, filed on Mar. 11, 2014, entitled “SYSTEM AND METHODFOR AUGMENTED AND VIRTUAL REALITY”, which is hereby incorporated byreference in its entirety.

FIELD OF INVENTION

The present disclosure generally relates to systems and methodsconfigured to facilitate interactive virtual or augmented realityenvironments for one or more users.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so-called “virtual reality” (VR), “augmentedreality” (AR) experiences, and/or “mixed reality” experiences(hereinafter collectively referred to as “mixed reality” and/or “MR”),where digitally reproduced images or portions thereof are presented to auser in a manner where they seem to be, or may be perceived as, real. AVR scenario typically involves presentation of digital or virtual imageinformation without transparency to other actual real-world visualinput, whereas an AR or MR scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the real world around the user such that the digital or virtual image(e.g., virtual content) may appear to be a part of the real world.However, MR may integrate the virtual content in a contextuallymeaningful way, whereas AR may not.

Therefore, there is a need for an approach to manage and display virtualcontent in a mixed reality environment.

SUMMARY

In accordance with some embodiments, instead of managing and displayingvirtual content in a mixed reality environment on a one-at-a-time basisindependently by each application, each virtual content is rendered byits respective application into a bounded volume, which hereinafter maybe referred to as a “Prism.” Each Prism may have characteristics andproperties that allow an application, sometimes called a universeapplication, to manage and display the Prism in the mixed realityenvironment such that the universe application may manage the placementand display of the virtual content in the mixed reality environment bymanaging the Prism itself.

One embodiment is directed to a method for displaying virtual content ina 3D spatial environment, the method comprising 1) receiving, from anapplication, a request to display a virtual content in a 3D spatialenvironment, 2) creating a Prism for managing the display of the virtualcontent, wherein the Prism is a cubic and/or rectangular volume of spaceconfigured to bound the virtual content inside a boundary of the Prism,3) receiving the virtual content from the application, 4) rendering thevirtual content inside the boundary of the Prism, associating the Prismto an object in the 3D spatial environment based at least in part on auser input, and 6) anchoring the Prism into the 3D spatial environment.

In one or more embodiments, borders of the boundary of the Prism are notdisplayed. The 3D spatial environment may be a physical real worldenvironment of a user. The Prism may be created automatically having aset of functionalities. The set of functionalities may comprise aminimum/maximum size allowed for the Prism, and/or an aspect ratio forresizing the Prism. The set of functionalities may comprise anassociation between the Prism to the object in the 3D spatialenvironment. The application may render additional virtual content intoadditional Prisms, wherein each virtual content may be rendered into aseparate Prism.

In one or more embodiments, the Prism does not overlap with other Prismsin the 3D spatial environment. The Prism may comprise one or moreuniversal features to ensure different applications interactappropriately with one another, and/or one or more application-specificfeatures, wherein the one or more universal features and the one or moreapplication-specific features are selected from a list of pre-approvedoptions.

Another embodiment is directed to a display system for displayingvirtual content in a 3D spatial environment, the display system mayinclude an augmented reality head-mounted display system, and one ormore modules for processing data, wherein the one or more modules arestored in one or more memory, the one or more modules configured toperform 1) receiving, from an application, a request to display avirtual content in a 3D spatial environment, 2) creating a prism,wherein the prism is a volume of space configured to bound the virtualcontent inside a boundary of the prism, 3) receiving the virtual contentfrom the application, 4) rendering the virtual content inside theboundaries of the prism, and 5) associating the prism to an object inthe 3D spatial environment.

In one or more embodiments, borders of the boundary of the Prism are notdisplayed. The 3D spatial environment may be a physical real worldenvironment of a user. The Prism may be created automatically having aset of functionalities. The set of functionalities may comprise aminimum/maximum size allowed for the Prism, and/or an aspect ratio forresizing the Prism. The set of functionalities may comprise anassociation between the Prism to the object in the 3D spatialenvironment. The application may render additional virtual content intoadditional Prisms, wherein each virtual content may be rendered into aseparate Prism.

In one or more embodiments, the Prism does not overlap with other Prismsin the 3D spatial environment. The Prism may comprise one or moreuniversal features to ensure different applications interactappropriately with one another, and/or one or more application-specificfeatures, wherein the one or more universal features and the one or moreapplication-specific features are selected from a list of pre-approvedoptions.

Another embodiment is directed to a method for starting a mixed realitysystem, the method may include determining a current location of a user,retrieving one or more Prisms previously deployed at the currentlocation, restoring the one or more Prisms at the current location ofthe user, and displaying the one or more Prisms restored at the currentlocation of the user.

In one or more embodiments, a Prism is a cubic and/or rectangular volumeof space that virtual content from an application is displayed into,wherein the may render into more than one Prism. In other words, in someembodiments, a single application may correspond to more than one Prism.In some embodiments, a single application corresponds to a single prism.A Prism represents a sub-tree of a multi-application scene graph for thecurrent location of the user. Retrieving the one or more Prismspreviously deployed at the current location of the user may compriseretrieving instance data for the one or more Prisms, from an externaldatabase for example, and reconstructing a local Prism database with theinstance data for the one or more Prisms, wherein the instance data foreach Prism include a data structure of Prism properties defining thePrism, the Prism properties comprising at least one of a location, anorientation, an extent width, an extent height, an extent depth, ananchor type, and/or an anchor position, wherein the instance data foreach Prism include key value pairs of application specific propertiescomprising state information of virtual content previously rendered intothe Prism by an application. In some embodiments, data is stored locallyand an external database is not needed.

In one or more embodiments, restoring the one or more Prisms compriseslaunching respective applications corresponding to the one or morePrisms previously deployed at the current location, creating one or morenew Prisms corresponding to the one or more Prisms previously deployed,and rendering respective virtual content into the one or more newPrisms.

In one or more embodiments, further comprising updating the local Prismdatabase of the user with updated Prism instance data as the userinteracts with the one or more Prisms, and synchronizing the local Prismdatabase with the external database.

Some embodiments are directed to a method for managing applicationstates of virtual content in a mixed reality system, the method mayinclude segmenting a 3D volume into a volumetric grid. The method mayalso include determining a first location of a user within thevolumetric grid. The method may further include determining a secondlocation of an application within the 3D volume. The method may alsoinclude calculating a distance between the user and the applicationwithin the 3D volume. The method may additionally include modifying astate of the application based at least in part on the distancecalculated between the user and the application.

Another embodiment is directed to a method for managing applicationstates of virtual content in a mixed reality system, the methodcomprising recording a spatial position of one or more applications in avolumetric grid, the one or more applications providing contentdisplayed inside one or more respective Prisms, the volumetric gridcorresponding to a coarse representation of a physical environment in anx, y, and z axis. Identifying a cell of the volumetric grid comprisingapplications located within the cell, wherein a width of the cell isequal to or larger than a radius of an active zone. Determining adistance of known positions of each application within the cell andneighboring cells when a user using a mixed reality device moves withinthe cell. And modifying a state of an application based at least in parton a distance between the mixed reality device and each applicationwithin the cell and neighboring cells.

In one or more embodiments, the radius of the active zone defines acircular/spherical area around the user using the mixed reality device,wherein the user may be at the center of the circle/sphere. In someembodiments, modifying the state of the application may be based, atleast in part, on whether the application is occluded by anotherapplication. In some embodiments, modifying the state of the applicationis based, at least in part, on a head pose of the user. A head pose ofthe user is a measurement of a location and/or orientation of a user'shead. The head pose can be used to render a scene to match a user'sdynamically changing head location and orientation to provide anincreased sense of immersion in a virtual/augmented/mixed space. In someembodiments, the head pose may be determined, at least in part, by aninertial measurement unit mounted on the head mounted display system, orthe user's head, although other suitable methods may also be used. Thedistance of known positions of each application within the cell may bedetermined only for the cell the user using the mixed reality device isin and the neighboring cells.

In one or more embodiments, there may be a buffer zone around anexterior of the active zone, wherein the buffer zone preventsintermittent or rapid changes to the state of the applications.

Another embodiment is directed to a method for launching an applicationfrom a launcher menu for displaying virtual content in a mixed realitysystem, the method comprising receiving a request for starting anapplication. Creating, by a universe application, a Prism for displayingand managing virtual content, wherein the Prism is a volumetric displayspace having boundaries for the application to render the virtualcontent into. Starting, by the Prism, the application through alifecycle service. Determining, by the Prism, a unique identifier (UID)of the application through the package manager service. Registering, bythe application, a listener with the universe application. Determining,by the universe application, the UID of the application. Associating, bythe universe application, the Prism to the listener. Assigning, by theuniverse application, the Prism to the application using the listener ofthe application. And placing the Prism in a sub-set of a 3D displayablespace of the mixed reality system.

Some embodiments are directed to a method for opening and placing anapplication in an augmented reality environment, the method may includereceiving a first user input from a user indicating an interest incontent. The method may also include launching an application togenerate the content. The method may further include creating a minidisplay volume of a 3D display volume managing unit, wherein a pagepreview is displayed in the mini display volume, wherein the minidisplay volume managing unit is created simultaneously with thelaunching of the application. The method may further include receiving asecond user input indicating a movement of the mini display volume. Themethod may also include receiving a third user input indicating aplacement of the mini display volume at a location in the augmentedreality environment, and expanding the 3D display volume managing unitin place of the mini display volume at the location, the 3D displayvolume managing unit displaying the content fully loaded within the 3Ddisplay volume managing unit.

In one or more embodiments, the first user input may be a cursormovement over a link on a web page, wherein the second user input is aselection of the link, and movement of the mini display volume. The minidisplay volume may be an initial default size of the 3D display volumemanaging unit. The content may be loaded into the mini display volumewhile the user is moving and placing the mini display volume. Thelocation may be fixed to an object in the augmented reality environment,wherein the object is the user.

Some embodiments may be directed to a method for managing virtualcontent, the method may include receiving content from a contentgenerating application. The method may also include displaying thecontent in a 3D spatial environment by a universe application. Themethod may further include constantly managing the display of thecontent in the 3D spatial environment by the universe application.

Some embodiments may be directed to a method that includes accessing ascene graph for a scene, wherein the scene graph comprises one or moretransform trees, each tree comprising a plurality of nodes. The methodmay also include adding a tag to one or more nodes from a plurality ofnodes within the one or more transform trees, wherein the one or morenodes tagged form a transform group, wherein the one or more nodestagged comprise a first tagged node and a second tagged node. The methodmay further include moving the first tagged node, wherein moving thefirst tagged node causes the second tagged node to move.

Another embodiment is directed to a method of displaying virtual contentin a 3D shared space, the method may include generating, by a firstapplication, virtual content in the 3D shared space, and displaying, bya second application, the virtual content generated by the firstapplication, the first application and the second application beingdifferent applications.

Some embodiments may be directed to a method for assigning to a Prismuniversal features and application selected features from a list ofpre-approved options for configurations of display customizations by anapplication. Another embodiment is directed to a method for displayingvirtual content into one or more Prisms, wherein the one or more Prismsdo not overlap with one another. Another embodiment is directed to amethod for changing a state of a Prism based at least in part on arelative position and location of the Prism to a user. Anotherembodiment is directed to a method for managing content creation in anapplication and managing content display in a separate application.Another embodiment is directed to a method for opening an applicationthat will provide content into a Prism while simultaneously placing thePrism in a mixed reality environment.

Some embodiments may be directed to a method for assigning location,orientation, and extent data to a Prism for displaying virtual contentwithin the Prism, the virtual content is 3D virtual content.

In one or more embodiment, the location is a coordinate of an anchorposition of the Prism in a mixed reality environment. The extent datadefines a size of the Prism.

Some embodiments may be directed to a method for pinning a launcherapplication to a real world object within a mixed reality environment.

In one or more embodiments, the pinned launcher application launchescontent of the application within a Prism in a same location as thepinned launcher application.

Some embodiments may be directed to a method for assigning a behaviortype to each Prism, the behavior type comprising at least one of a worldlock, a billboard, an edge billboard, a follow headlock, a follow basedon external sensor, or a fade (described in more detail below). Someembodiments are directed to a method for identifying a most used contentspecific to a placed location of a launcher application. Someembodiments are directed to a method for displaying favoriteapplications, by a placed launcher application, the favoriteapplications based at least in part on context relative to a location ofthe placed launcher.

Additional and other objects, features, and advantages of the disclosureare described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of preferred embodimentsof the present disclosure, in which similar elements are referred to bycommon reference numerals. In order to better appreciate how theabove-recited and other advantages and objects of the present disclosureare obtained, a more particular description of the present disclosurebriefly described above will be rendered by reference to specificembodiments thereof, which are illustrated in the accompanying drawings.Understanding that these drawings depict only typical embodiments of thedisclosure and are not therefore to be considered limiting of its scope,the disclosure will be described and explained with additionalspecificity and detail through the use of the accompanying drawings.

The drawings use like reference numerals to identify like elements. Aletter after a reference numeral, such as “120a,” indicates that thetext refers specifically to the element having that particular referencenumeral. A reference numeral in the text without a following letter,such as “120,” refers to any or all of the elements in the drawingsbearing that reference numeral (e.g. “120” in the text refers toreference numerals “120a” and/or “120b” in the drawings).

FIG. 1 shows an example user physical environment and systemarchitecture for managing and displaying virtual content in a mixedreality system, according to some embodiments.

FIG. 2 shows a system architecture for managing and displaying virtualcontent in a mixed reality system, according to some embodiments.

FIG. 3 shows a Prism/bounding volume, according to some embodiments.

FIG. 4 shows a launcher menu for launching applications to displayvirtual content, according to some embodiments.

FIG. 5A-5B shows a panel carousel change, according to some embodiments.

FIG. 6 shows a flowchart for an approach for starting a mixed realitysystem, according to some embodiments.

FIG. 7 shows a flowchart for an approach for displaying a virtualcontent in a mixed reality environment, according to some embodiments.

FIG. 8 is a diagram for managing application states, according to someembodiments.

FIG. 9 shows a flowchart for managing an application state relative to auser location, according to some embodiments.

FIG. 10A illustrates a tree node of a scene graph, according to someembodiments.

FIGS. 10B-10AA illustrate various transform trees and group trees,according to some embodiments.

FIG. 11 shows a view of a placed launcher, according to someembodiments.

FIG. 12 shows types of content that may be displayed while a launcher isin an idle/sleeping state, according to some embodiments.

FIG. 13 shows a Secondary UI volume, according to some embodiments.

FIG. 14 shows an example of body dynamics, according to someembodiments.

FIG. 15 shows different types of body dynamics, according to someembodiments.

FIG. 16 shows a flowchart for simultaneously launching and placing anapplication in a mixed reality environment, according to someembodiments.

FIG. 17 is a block diagram of an illustrative computing system 1400suitable for implementing one or more of the embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The present disclosure is directed to managing and displaying virtualcontent in a mixed reality system. Instead of allowing multipleapplications to manage and display virtual content in a mixed realitysystem independently of one another, embodiments of the presentdisclosure disclose a universe application that manages (e.g. how andwhere) virtual content to be displayed and managed in the mixed realitysystem using 3D windows called Prisms.

This disclosure provides a description of an illustrative mixed realitysystem with which some embodiments of the disclosure may be practiced,followed by a description of one or more embodiments of processes andmechanisms to manage and display virtual content in the mixed realitysystem.

The description that follows pertains to an illustrative mixed realitysystem with which the disclosure may be practiced. However, it is to beunderstood that the disclosure also lends itself to applications inother types of AR and virtual reality (VR) systems, and therefore thedisclosure is not to be limited to only the illustrative systemdisclosed herein.

FIG. 1 , shows an example user physical environment and systemarchitecture for managing and displaying virtual content in a mixedreality system. The representative environment 100 includes a user'slandscape 110 as viewed by a user 103 through a head-mounted system 160.The user's landscape 110 is a 3D view of the world where user-placedcontent may be composited on top of the real world. The representativeenvironment 100 further includes accessing a universe application 130via a processor 170 operatively coupled to a network (not shown).Although the processor 170 is shown as an isolated component separatefrom the head-mounted system 160, in an alternate embodiment, theprocessor 170 may be integrated with one or more components of thehead-mounted system 160, and/or may be integrated into other systemcomponents within the representative environment 100 such as, forexample, a network to access a computing network (not shown) andexternal storage device(s) 150. In some embodiments, the processor 170may not be connected to a network. The processor 170 may be configuredwith software (e.g., a universe application 130) for receiving andprocessing information such as video, audio, and/or other data (e.g.depth camera data) received from the head-mounted system 160, a localstorage device 137, application(s) 140, a computing network, and/orexternal storage device(s) 150.

The universe application 130 may be a 3D windows manager that isanalogous to a 2D windows manager running on, for example, a desktopcomputer for managing 2D windows displayed on the display screen of thedesktop computer. However, the universe application 130 (hereinafter maybe referred to as “the Universe”) manages the creation, placement anddisplay of virtual content 115 in a 3D spatial environment, as well asinteractions between a plurality of virtual content 115 displayed in auser's landscape 110. Virtual content 115 from applications 140 arepresented to users 103 inside of one or more 3D window displaymanagement units such as bounded volumes and/or 3D windows, hereinaftermay be referred to as Prisms 113.

A bounded volume/3D window/Prism 113 may be a rectangular, cubic,cylindrical, or any other shape volume of space that may be positionedand oriented in space. A Prism 113 may be a volumetric display spacehaving boundaries for content (e.g., virtual content) to berendered/displayed into, wherein the boundaries are not displayed. Insome embodiments, the boundaries may be displayed. The Prism 113 maypresent a standard base level of interaction and control over anapplication's content and its placement. The Prism 113 may represent asub-tree of a multi-application scene graph, which may be embeddedinside of the Universe, or may be external to but accessed by theUniverse. A scene graph is a general data structure commonly used byvector-based graphics, editing applications and modern gaming software,which arranges the logical and often (but not necessarily) spatialrepresentation of a graphical scene. A scenegraph may be considered adata-structure that defines how content is positioned and transformedrelative to each other within its structure. Application(s) 140 aregiven instances of Prisms 113 to place content within. Applications mayrender 2D/3D content within a Prism 113 using relative placementalgorithms and arbitrary transforms, but the Universe may stillultimately be in charge of gross interaction patterns such as contentextraction. Multiple applications may render to the Universe via thePrisms 113, with process boundaries separating the Prisms 113. There maybe n number of bounded volumes/Prisms 113 per application process, butthis is explicitly an n:1 relationship such that only one process foreach application may be running for each bounded volume/Prism 113, butthere may be a number of m processes running, each with their ownbounded volume/Prism 113.

The Universe operates using a Prism/distributed scene graph approach for2D and/or 3D content. A portion of the Universe's scene graph isreserved for each application to render to. Each interaction with anapplication, for example the launcher menu, the landscape, orbody-centric application zones (all described in more detail below) maybe done through a multi-application scene graph. Each application isallocated 1 to n rectangular Prisms that represent a sub-tree of thescene graph. Prisms are not allocated by the client side applications,but instead are created through the interaction of the user inside ofthe Universe, for example when the user opens a new application in thelandscape by clicking a button on a controller. In some embodiments, anapplication can request a Prism from the Universe, but the request maybe denied. In some embodiments, if an application requests and isallowed a new Prism, the application may only transform the new Prismrelative to one of its other Prisms.

The Universe encloses virtual content 115 from application(s) 140 inobjects called Prisms 113. Each application process or instance mayrender its virtual content into its own individual Prism 113 or set ofPrisms. The Universe manages a world space, sometimes called alandscape, where Prisms 113 are displayed. In some embodiments, theUniverse provides the ability to attach applications to walls andsurfaces, place Prisms at an arbitrary location in space, register themwith the mixed reality system's world database, and/or control sharingof content between multiple users of the mixed reality system.

In some embodiments, the purpose of the Prisms 113 is to providebehaviors and control over the rendering and display of the content.Much like a 2D display, where a window may be used to define location,menu structures, and display of 2D content within a 2D window, with 3Dvirtual display, the Prism allows the mixed reality system (e.g., theUniverse) to wrap control relating to, for example, content locations,3D window behavior, and/or menu structures around the display of 3Dcontent. For example, controls may include at least placing the virtualcontent in a particular location in the user's landscape 110, removingthe virtual content from the landscape 110, copying the virtual contentand/or placing the copy in a different location, etc. In someembodiments, Prisms may be created and destroyed by the user and onlythe user. This may be done explicitly to help control abuse of theinterfaces provided and to help the user maintain control of the user'scontent. Additionally, in some embodiments, application(s) 140 do notknow where their volumes are placed in the landscape—only that theyexist. In some embodiments, applications may request one or more Prisms,and the request may or may not be granted. After the new Prism iscreated, the user may change the position, and/or the application mayautomatically position the new Prism relative to a currently existingPrism associated with the application. In some embodiments, eachapplication 140 making use of the Universe service to render 3D content(e.g. composited 3D content) into the Universe process may be requiredto first register a listener with the Universe. This listener may beused to inform the application 140 of creation and destruction ofrendering Prisms, based upon user movement and user interaction withthose Prisms. A listener is an interface object that receives messagesfrom an inter-process communication system. For example, in the Androidoperating system, a listener is an object that receives messages throughan Android Binder interface. However, any IPC system may be used suchthat a Binder is not always used.

In some embodiments, Prisms may be created from the followinginteractions: (1) The user has extracted content from an extractablenode (disclosed further below); (2) The user has started an applicationfrom the launcher; (3) The user has downloaded a nearby passable worldmap tile that includes a placed instance of an application that the userhas permission to see; (4) The user has downloaded a nearby passableworld map tile that includes an object that the passable world objectrecognizer infrastructure has detected, that a given application mustrender content for; and/or (5) The user has triggered a dispatch fromanother application that must be handled in a different application. Insome embodiments, a passable world model allows a user to effectivelypass over a piece of the user's world (e.g., ambient surroundings,interactions, etc.) to another user.

Extractable Content is content inside a Prism (including but not limitedto an icon, 3D icon, word in a text display, and/or image) that can bepulled out of the Prism using an input device and placed in thelandscape. For example, a Prism might display a web page showing arunning shoe for sale. To extract the running shoe, the shoe can beselected and “pulled” with an input device. A new Prism would be createdwith a 3D model representing the shoe, and that Prism would move out ofthe original Prism and towards the user Like any other Prism, the usermay use an input device to move, grow, shrink or rotate the new Prismcontaining the shoe in the 3D space of the landscape. An ExtractableNode is a node in the Prism's scene graph that has been tagged assomething that can be extracted. In the Universe, to extract contentmeans to select an extractable node, and use an input device to pull thecontent out of the Prism. The input to initiate this pull could beaiming a 6dof pointing device at extractable content and pulling thetrigger on the input device.

Each user's respective individual mixed reality system (e.g., mixedreality devices) captures information as the user passes through orinhabits an environment, which the mixed reality system processes toproduce a passable world model. More details regarding a passable worldare described in U.S. patent application Ser. No. 14/205,126, filed onMar. 11, 2014, entitled “SYSTEM AND METHOD FOR AUGMENTED AND VIRTUALREALITY”, which has been previously incorporated by reference. Theindividual mixed reality system may communicate or pass the passableworld model to a common or shared collection of data, referred to as thecloud. The individual mixed reality system may communicate or pass thepassable world model to other users, either directly or via the cloud.The passable world model provides the ability to efficiently communicateor pass information that essentially encompasses at least a field ofview of a user. In one embodiment, the system uses the pose andorientation information, as well as collected 3D points described abovein order to create the passable world. In some embodiments, the passableworld model allows the user the ability to integrate content (e.g.,virtual and/or physical content) with the real world. A passable worldsystem may include one or more mixed reality systems or mixed realityuser devices that are able to connect to a cloud network, a passableworld model, a set of object recognizers, and a database (e.g., externaldatabase 150). The passable world model may be configured to receiveinformation from the mixed reality user devices and also transmit datato them through the network. For example, based on the input from auser, a piece of the passable world may be passed on from one user toanother user. The passable world model may be thought of as a collectionof images, points and other information (e.g. real world information)based on which the mixed reality system is able to construct, update andbuild the virtual world on the cloud, and effectively pass pieces of thevirtual world to various users. For example, a set of real world pointscollected from a mixed reality user device may be collected in thepassable world model. Various object recognizers may crawl through thepassable world model to recognize objects, tag images, etc., and attachsemantic information to the objects. The passable world model may usethe database to build its knowledge of the world, attach semanticinformation, and store data associated with the passable world.

In the case of a Prism that is visible to the user but whose controllingapplication is not currently installed, the Universe may render atemporary placeholder for that application that, when interacted with,redirects the user to the application store page for that application.

In some embodiments, Prisms may be destroyed in similar interactions:(1) The user has walked far enough from a passable world map tile thatthe placed instance of an application has been unloaded (i.e. removed)from volatile memory; (2) The user has destroyed a placed instance of anapplication; and/or (3) An application has requested that a Prism beclosed.

In some embodiments, if no Prisms for an application are visible and/orloaded, then the process associated with those Prisms may be paused orended. Once a placed Prism for that application is visible again, theprocess may be restarted. Prisms may also be hidden, but, in someembodiments, this may only happen at the behest of the Universe and theuser. In some embodiments, multiple Prisms may be placed at the sameexact location. In such embodiments, the Universe may only show oneinstance of a placed Prism in one place at a time, and manage therendering by hiding the visibility of a Prism (and its associatedcontent) until a user interaction is detected, such as the user “swipes”to the next visible element (e.g., Prism) in that location.

In some embodiments, each Prism 113 may be exposed to the application140 via a volume listener interface with methods for accessingproperties of the Prism 113 and registering content in a scene graphsub-tree for shared resources such as meshes, textures, animations, andso on.

In some embodiments, since the application 140 does not know where agiven Prism 113 is placed in 3D space, the volume listener interface mayprovide accessor methods to a set of hints that help to define where thegiven Prism is present in the Universe, for example hand centric, stuckin the landscape, Body Centric, etc. These properties additionallyspecify expected behavior of the Prisms, and may be controlled in alimited fashion either by the user, the application 140, or theUniverse. A given Prism can be positioned relative to another Prism thatan application owns. Applications can specify that Prisms should snaptogether (two sides of their bounding volumes touch) while Prisms fromthat application are being placed. Additionally, Prisms may provide anAPI for key-value data storage. Some of these key-value pairs are onlywritable by privileged applications.

In some embodiments, application(s) 140 are client software applicationsthat provide content that is to be displayed to the user 103 in theuser's landscape 110. For example, an application 140 may be a videostreaming application, wherein video data may be streamed to the user tobe displayed on a 2D planar surface. As another example, an application140 may be a Halcyon application that provides 3D imaging of physicalobjects that may denote a period of time in the past that wasidyllically happy and peaceful for the user. Application 140 providesthe content that a user may want to include in the user's landscape 110.The Universe via the Prisms 113 manages the placement and management ofthe content that is generated by application 140.

When a non-immersive application is executed/launched in the user'slandscape 110, its content (e.g., virtual content) is rendered inside ofa Prism 113. A non-immersive application may be an application that isable to run and/or display content simultaneously with one or more otherapplications in a shared 3D environment. Although the virtual contentmay be contained within the Prism, a user may still interact with thevirtual content, such as, for example, hovering over an object, clickingon it, etc. The Prism 113 may also bound application 140's displayedcontent so different applications 140 do not interfere with each otheror other objects in the user's landscape 110. Prisms 113 may alsoprovide a useful abstraction for suspending, pausing, and/or minimizingvirtual content from application(s) 140 that are out of view or too faraway from the user.

The Prisms 113 may be anchored/attached/pinned to various objects withina user's landscape 110, including snapping or anchoring to anotherPrism. For example, Prism 113 a, which displays virtual content 115(e.g., a video 115 a from a video streaming application), may beanchored to a vertical wall 117 a. As another example, Prism 113 b,which displays a 3D tree 115 b from a Halcyon application, is shown inFIG. 1 to be anchored to a table 117 b. Furthermore, a Prism 113 may beanchored relative to a user 103 (e.g., body-centric), wherein the Prism113 which displays virtual content 115 may be anchored to a user's body,such that as the user's body moves, the Prism 113 moves relative to themovement of the user's body. A body-centric content may be applicationcontent such as planes, meshes, etc. that follow the user and remainpositionally consistent with the user. For example, a small dialog boxthat follows the user around but exists relative to the user's spinerather than the landscape 110. Additionally, a Prism 113 may also beanchored to a virtual object such as a virtual display monitor displayedwithin the user's landscape 110. The Prism 113 may be anchored indifferent ways, which is disclosed below.

The Universe may include a local database 137 to store properties andcharacteristics of the Prisms 113 for the user. The stored Prisminformation may include Prisms activated by the user within the user'slandscape 110. Local database 137 may be operatively coupled to anexternal database 150 that may reside in the cloud or in an externalstorage facility. External database 150 may be a persisted database thatmaintains information about the mixed reality environment of the userand of other users.

For example, as a user launches a new application to display virtualcontent in the user's physical environment, the local database 137 maystore information corresponding to a Prism that is created and placed ata particular location by the Universe, wherein an application 140 mayrender content into the Prism 113 to be displayed in the user'slandscape 110. The information corresponding to the Prism 113, virtualcontent 115, and application 140 stored in the local database 137 may besynchronized to the external database 150 for persistent storage.

In some embodiments, the persisted storage may be important because whenthe mixed reality system is turned off, data stored in the localdatabase 137 may be erased, deleted, or non-persisted. Thus, when a userturns on the mixed reality system, the Universe may synchronize with theexternal database 150 to retrieve an instance of the local database 137corresponding to the user 103 and the user's landscape 110 prior to themixed reality system being turned off. The local database 137 may be aninstance of the external database 150, wherein the instance of the localdatabase 137 includes information pertinent to the user 103 and theuser's current environment. The external database 150 may additionallystore instances of local databases of other users, multiple users, thesame user over time, and/or other environments. The external database150 may contain information that is used to manage and share virtualcontent between multiple users of the mixed reality system, whereas thelocal database 137 stores and maintains information corresponding to theuser 103.

The Universe may create a Prism 113 for application 140 each timeapplication(s) 140 needs to render virtual content 115 onto a user'slandscape 110. In some embodiments, the Prism 113 created by theUniverse allows application 140 to focus on rendering virtual contentfor display while the Universe focuses on creating and managing theplacement and display of the Prism 113 having the virtual content 115displayed within the boundaries of the Prism by the application 140.

Each virtual content 115 rendered by an application 140, displayed inthe user's landscape 110, may be displayed within a single Prism 113.For example, if an application 140 needs to render two virtual contents(e.g., 115 a and 115 b) to be displayed within a user's landscape 110,then application 140 may render the two virtual contents 115 a and 115b. Since virtual contents 115 include only the rendered virtualcontents, the Universe may create Prisms 113 a and 113 b to correspondwith each of the virtual content 115 a and 115 b, respectively. ThePrism 113 may include 3D windows management properties andcharacteristics of the virtual content 115 to allow the Universe tomanage the virtual content 115 inside the Prism 113 and the placementand display of the Prism 113 in the user's landscape 110.

The Universe may be the first application a user 103 sees when the user103 turns on the mixed reality device. The Universe may be responsiblefor at least (1) rendering the user's world landscape; (2) 2D windowmanagement of planar applications and 3D windows (e.g., Prisms)management; (3) displaying and executing the application launcher menu;(4) allowing the user to place virtual content into the user's landscape110; and/or (5) managing the different states of the display of thePrisms 113 within the user's landscape 110.

The head-mounted system 160 may be a mixed reality head-mounted systemthat includes a display system (e.g., a user interface) positioned infront of the eyes of the user 103, a speaker coupled to the head-mountedsystem and positioned adjacent the ear canal of the user, a user-sensingsystem, an environment sensing system, and a processor (all not shown).The head-mounted system 160 presents to the user 103 the display system(e.g., user interface) for interacting with and experiencing a digitalworld. Such interaction may involve the user and the digital world, oneor more other users interfacing the representative environment 100, andobjects within the digital and physical world.

The user interface may include viewing, selecting, positioning andmanaging virtual content via user input through the user interface. Theuser interface may be at least one or a combination of a hapticsinterface devices, a keyboard, a mouse, a joystick, a motion capturecontroller, an optical tracking device, an audio input device, asmartphone, a tablet, or the head-mounted system 160. A hapticsinterface device is a device that allows a human to interact with acomputer through bodily sensations and movements. Haptics refers to atype of human-computer interaction technology that encompasses tactilefeedback or other bodily sensations to perform actions or processes on acomputing device.

An example of a haptics controller may be a totem (not shown). In someembodiments, a totem is a hand-held controller that tracks its positionand orientation relative to the headset 160. In this example, the totemmay be a six degree-of-freedom (six DOF) controller where a user maymove a Prism around in altitude and azimuth (on a spherical shell) bymoving the totem up or down. In some embodiments, to move the objectcloser or farther away, the user may use the joystick on the totem to“push” or “pull” the Prism, or may simply move the totem forward orbackward. This may have the effect of changing the radius of the shell.In some embodiments, two buttons on the totem may cause the Prism togrow or shrink. In some embodiments, rotating the totem itself mayrotate the Prism. Other totem manipulations and configurations may beused, and should not be limited to the embodiments described above.

The user-sensing system may include one or more sensors 162 operable todetect certain features, characteristics, or information related to theuser 103 wearing the head-mounted system 160. For example, in someembodiments, the sensors 162 may include a camera or opticaldetection/scanning circuitry capable of detecting real-time opticalcharacteristics/measurements of the user 103 such as, for example, oneor more of the following: pupil constriction/dilation, angularmeasurement/positioning of each pupil, sphericity, eye shape (as eyeshape changes over time) and other anatomic data. This data may provide,or be used to calculate information (e.g., the user's visual focalpoint) that may be used by the head-mounted system 160 to enhance theuser's viewing experience.

The environment-sensing system may include one or more sensors 164 forobtaining data from the user's landscape 110. Objects or informationdetected by the sensors 164 may be provided as input to the head-mountedsystem 160. In some embodiments, this input may represent userinteraction with the virtual world. For example, a user (e.g., the user103) viewing a virtual keyboard on a desk (e.g., the table 188) maygesture with their fingers as if the user were typing on the virtualkeyboard. The motion of the fingers moving may be captured by thesensors 164 and provided to the head-mounted system 160 as input,wherein the input may be used to change the virtual world or create newvirtual objects.

The sensors 164 may include, for example, a generally outward-facingcamera or a scanner for capturing and interpreting scene information,for example, through continuously and/or intermittently projectedinfrared structured light. The environment-sensing system may be usedfor mapping one or more elements of the user's landscape 110 around theuser 103 by detecting and registering one or more elements from thelocal environment, including static objects, dynamic objects, people,gestures and various lighting, atmospheric and acoustic conditions, etc.Thus, in some embodiments, the environment-sensing system may includeimage-based 3D reconstruction software embedded in a local computingsystem (e.g., the processor 170) and operable to digitally reconstructone or more objects or information detected by the sensors 164.

In some embodiments, the environment-sensing system provides one or moreof the following: motion capture data (including gesture recognition),depth sensing, facial recognition, object recognition, unique objectfeature recognition, voice/audio recognition and processing, acousticsource localization, noise reduction, infrared or similar laserprojection, as well as monochrome and/or color CMOS sensors (or othersimilar sensors), field-of-view sensors, and a variety of otheroptical-enhancing sensors. It should be appreciated that theenvironment-sensing system may include other components other than thosediscussed above.

As mentioned above, the processor 170 may, in some embodiments, beintegrated with other components of the head-mounted system 160,integrated with other components of the system of the representativeenvironment 100, or may be an isolated device (wearable or separate fromthe user 103) as shown in FIG. 1 . The processor 170 may be connected tovarious components of the head-mounted system 160 through a physical,wired connection, or through a wireless connection such as, for example,mobile network connections (including cellular telephone and datanetworks), Wi-Fi, Bluetooth, or any other wireless connection protocol.The processor 170 may include a memory module, integrated and/oradditional graphics processing unit, wireless and/or wired internetconnectivity, and codec and/or firmware capable of transforming datafrom a source (e.g., a computing network, and the user-sensing systemand the environment-sensing system from the head-mounted system 160)into image and audio data, wherein the images/video and audio may bepresented to the user 103 via the user interface (not shown).

The processor 170 handles data processing for the various components ofthe head-mounted system 160 as well as data exchange between thehead-mounted system 160 and the software applications such as theUniverse, the external database 150, etc. For example, the processor 170may be used to buffer and process data streaming between the user 103and the computing network, including the software applications, therebyenabling a smooth, continuous and high fidelity user experience. Theprocessor 170 may be configured to execute a set of program codeinstructions. The processor 170 may include a memory to hold the set ofprogram code instructions, in which the set of program code instructionscomprises program code to display virtual content within a subset ofavailable 3D displayable space by displaying the virtual content withina volumetric display space, wherein boundaries of the volumetric displayspace are not displayed. In some embodiments, the processor may be twoor more processors operatively coupled.

In some embodiments, the mixed reality system may be configured toassign to a Prism universal features and applicationselected/application-specific features from a list of pre-approvedoptions for configurations of display customizations by an application.For example, universal features ensure different applications interactwell together. Some example of universal features may include max/minsize, no overlapping Prisms (excluding temporary overlap from collisionbehavior), no displaying content outside the boundaries of the Prism,applications need permission from user if the application wants toaccess sensors or sensitive information. Applicationselected/application-specific features enable optimized applicationexperiences. Application-selected/application-specific features mayinclude max/min size (within limits from the system), default size(within limits from the system), type of body dynamic (e.g., none/worldlock, billboard, edge billboard, follow/lazy headlock, follow based onexternal sensor, fade—discussed below), child Prism spawn location,child headpose highlight, child Prism relational behavior, on surfacebehavior, independent transformation control, resize vs. scale, idlestate timeout, collision behavior, permission/password to accessapplication, etc. In another embodiment, the mixed reality system may beconfigured to display virtual content into one or more Prisms, whereinthe one or more Prisms do not overlap with one another, in someembodiments. In some embodiments, one or more Prisms may overlap inorder to provide specific interactions. In some embodiments, one or morePrisms may overlap, but only with other Prisms from the sameapplication. In another embodiment, the mixed reality system may beconfigured to change a state of a Prism based at least in part on arelative position and location of the Prism to a user. In anotherembodiment, the mixed reality system may be configured to manage contentcreation in an application and manage content display in a separateapplication. In another embodiment, the mixed reality system may beconfigured to open an application that will provide content into a Prismwhile simultaneously placing the Prism in a mixed reality environment.

In some embodiments, the mixed reality system may be configured toassign location, orientation, and extent data to a Prism for displayingvirtual content within the Prism, where the virtual content is 3Dvirtual content. In some embodiments, the mixed reality system may beconfigured to pin a launcher application to a real world object within amixed reality environment. In some embodiments, the mixed reality systemmay be configured to assign a behavior type to each Prism, the behaviortype comprising at least one of a world lock, a billboard, an edgebillboard, a follow headlock, a follow based on external sensor, or afade (described below in more detail). In some embodiments, the mixedreality system may be configured to identify a most used content or anapplication that is specific to a placed location of a launcherapplication, and consequently re-order to the applications from most toleast frequently used, for example. In another embodiment, the mixedreality system may be configured to display favorite applications at aplaced launcher application, the favorite applications based at least inpart on context relative to a location of the placed launcher.

FIG. 2 shows a system architecture for managing and displaying virtualcontent in a mixed reality system, according to some embodiments. System200 includes a Universe 130, application 140, icon grid application 260,status bar app 270, social panel app 280, and store panel app 290. Theseapplications may represent the base level of applications on system 200,however, in some embodiments, more or fewer applications may be part ofsystem 200.

As discussed in FIG. 1 above, the Universe may be thought of as a 3Dwindows (e.g., Prisms) manager, analogous to a 2D windows manager thatmanages 2D windows in conventional computer desktop systems and such.FIG. 2 may provide further details of the Universe from FIG. 1 . Here,the universe application 130 may also include universe server 205,loader volumes 210, secondary UI volumes 220, universe client 225,launcher application 230, and universe server 205. Universe server 205may be a processing thread of the Universe in a multi-threadedprocessing environment for multi-parallel processing.

Loader volumes 210 are placeholder volumes that are displayed to a userwhile the Universe is creating a Prism for displaying virtual content inthe user's landscape 110. For example, when a user selects anapplication to display in the user's landscape 110 at a particularlocation, for example, on a vertical wall of the user's landscape 110,while the Universe is setting up the Prism and starting the applicationfor rendering the virtual content into the Prism, the Universe maydisplay a loader volume 210 with a default icon as a placeholder volumeto indicate to the user that the Universe is setting up the Prism fordisplay. Once the application finishes rendering the virtual contentinto the Prism for display in the user's landscape, the loader volume210 is replaced with the actual Prism containing the rendered virtualcontent.

In some embodiments, while the Universe is starting up an applicationfor displaying virtual content, the user 103 may move the loader volume210 to a desired different location. In some embodiments, the user maymove the loader volume 210 to a location that is different than thelocation of the loader volume/Prism that was initially selected. Oncethe universe is done creating the Prism and the application has renderedthe virtual content into the Prism, the Universe may replace the loadervolume 210, wherever the user may have placed the loader volume 210,with the Prism displaying the virtual content.

Secondary UI volume 220 is another Prism that may be created when aPrism 113 (e.g., its “parent Prism”) is created. The Secondary UI volume220 provides a universal interface of Prisms for users. For example, theSecondary UI volume 220 may be considered as window dressing because theSecondary UI volume 220 provides a mechanism to manage a Prism (e.g.,close/remove, share, follow, take a screenshot of the Prism's content,etc.). When a Prism is created, a Secondary UI volume 220 may be createdfor the Prism if the Prism is NOT part of the Launcher (Launcherapplications may not have Secondary UI volumes). The Secondary UI volume220 provides the space/volume to display graphical user interface iconssuch as close/remove, share, follow, screenshot, etc. for the user tointeract with and manage the Prism. The Secondary UI volume 220 isassociated to the parent Prism and may be grouped with the parent Prism.The Secondary UI volume 220 lifetime ends when the parent Prism lifetimeit is associated with ends.

In some embodiments, the Secondary UI volume 220 may have at least threestates: (1) Display nothing when the parent Prism is out of focus; (2)Display the component's “visible name” when the parent Prism is infocus; and (3) Display a “carousel” of application menu option iconswhen a specific user interaction is detected, for example, a home buttonof a handheld controller (e.g., a Totem, or other suitable userinteraction controllers) has been held for a certain number of seconds,wherein the carousel displays a collection of icons, one of which may bea large “X” icon for closing the Prism. In some embodiments, theSecondary UI volume 220 receives input via its parent Prism. In otherwords, the parent Prism may determine if the Secondary UI volume 220 isdisplaying its carousel, and if so the parent Prism redirects user inputto the Secondary UI. The carousel of the Secondary UI volume 220 isdisclosed below.

In some embodiments, the launcher may be the default “home” menu for themixed reality system. The launcher may bring together multiple panels ofcontent alongside a system status bar. Each panel may represent adifferent content type. Applications may be pulled from the launcher andpinned into the landscape for quick recall. The launcher itself may beplaced into the landscape for customization per location and/or forquick access.

Launcher 230 provides the user with the ability to launch newapplications into the user's landscape 110. The launcher 230 may be anapplication composed of a series of body-centric Prisms called panels.The panels may be vertically and horizontally scrollable and a user mayswitch between panels with a swiping motion, for example. In someembodiments, one panel may be visible at a time (e.g., a central panel),with its two neighboring panels visible as placeholder panels at itsside. When the user swipes to the next panel, the placeholder panels mayexpand to show the full panel. Panels may include an Icon Gridapplication 260, a Social panel 280, and a Store panel 290. In someembodiments, when the user swipes to the next panel, the panelsthemselves are not moved or changed, but instead, contents (e.g., icons)within the different panels may be animated in and out of the centralpanel (e.g., active panel). Furthermore, applications may be pulled fromthe launcher 230 and pinned into the user's landscape 110 forcustomization per location, discussed further below.

In some embodiments, an application 140 may communicate with theUniverse via a centralized rendering service client 250 on eachapplication 140. The centralized rendering service client 250 may be incommunication with a universe server 205 within the Universe 130. Thecentralized rendering service client 250 may be a client service of acentralized rendering system that allows application(s) 140 and otherapplications that generate content for display in the user's landscapeto communicate with the Universe via the universe server 205.

The universe server 205 may comprise a service of the centralizedrendering system that allows the Universe to communicate withapplications that provide the Universe content to be displayed in theuser's landscape. In some embodiments, the communication may comprisemore than rendering data, for example, input data, requesting a securityprivilege, requesting to show or hide the virtual keyboard, etc.

In some embodiments, the centralized rendering system may be a system ofhardware and software resources dedicated to receiving graphical datafrom multiple applications to be displayed on a single display (e.g., ina user's landscape in the mixed reality system). The centralizedrendering system combines graphical data from multiple applications 140into a “centralized” data structure, such as a scene graph, which may beused to render, to a display, a scene reflecting the graphical data fromthe multiple applications in a realistic and efficient manner. In orderto achieve the centralized rendering system, in some embodiments, anapplication may make changes to a local representation of the Prismcalled the Client Prism (e.g. Client Prism 215 from FIG. 2 ). Thesechanges may then be sent to the Universe Server 205 and stored in aServer Prism. The centralized rendering system may then render theupdated data in the Server Prism. The centralized rendering system mayhereinafter be referred to as the “Cali” or Kali” system. The Universemay be thought of as an enhanced version of the Cali Server, forexample, because the Universe can manage the Prisms in the real world.

In some embodiments, each application 140 that creates virtual content115 for the Universe communicates with the centralized rendering systemand the Universe via the centralized rendering service client 250(hereinafter may be referred to as a “Cali client”) installed on each ofthe respective application(s) 140. More information may be disclosed ina related Application Ser. No. 62/479,134 entitled “CENTRALIZEDRENDERING”, filed on Mar. 30, 2017, and which is hereby incorporated byreference in its entirety. The centralized rendering system improves theuser's experience by ensuring that virtual content from multipledifferent applications are properly analyzed and processed, ifnecessary, to ensure the virtual content are displayed in a realisticmanner to the user. In some embodiments, the Universe is an instance ofa Cali Server with additional functionality, such as managing Prisms. Insome embodiments, a Client Prism is an instance of a Cali Client Volumeand a Server Prism is an instance of a Cali Server Volume, withadditional functionality, such as the ability to bring up an App Optionsdisplay, to display a Loader Volume while the Prism is loading itscontent, to collide with other Prisms, and to be part of a TransformTree.

Client Prism 215 a and Client Prism 215 b comprise virtual content thatis generated by the application 140 and sent by the Cali Client 250 a tothe Universe Server 205 to be displayed in the user's landscape. In someembodiments, as the application 140 makes changes to the virtual content115 a and 115 b, the changes to the virtual content are communicatedfrom the Client Prism 215 to the Universe Server 205, and thatinformation is stored inside the Universe in the corresponding ServerPrism data structures 113. In some embodiments, the application 140 doesnot know where in the user's landscape a virtual content 115 a isdisplayed. The Universe may manage display location of the virtualcontent 115 a via the corresponding Server Prism 113 a that isassociated to the Client Prism 215 a (e.g., the virtual content 115 aafter it has been processed by the centralized rendering system). Theapplication 140 may request a new Prism by accessing Universe Server205. In some embodiments, the universe server 205 may be a softwaremodule in the Universe that communicates with centralized renderingservice client(s) 250 from applications that provide virtual content fordisplay in the user's landscape 110. For example, when a user wants tolaunch an application and display virtual content from the applicationin the user's landscape, the application may provide the virtual contentto the Universe via the centralized rendering service client from theapplication to the universe centralized rendering service on theUniverse to be displayed in a Prism that may be anchored in the user'slandscape.

In some embodiments, the icon grid application 260 may comprise a recentapplication section (not shown) and/or a general application section(not shown). The general application section comprises an iconrepresenting each application installed on the mixed reality system. Thegeneral application section may be initially populated with a call to aPackage Manager (not shown) to determine a list of installed packages.An icon is added for each application in each package. When the PackageManager notifies the Universe of package installation anduninstallation, the icon grid application 260 adjusts its iconsaccordingly. The Package Manager Service manages the installation ofapplications and maintains information about those applications such astheir names, icon graphics, security permissions, executable files anddata files.

The recent icon section may be initially reconstructed from a log ondisk, and then updated by calls from other services. The package namemay be logged to disk when a Lifecycle Service notifies the launcher ofan application start event, and when the Package Manager notifies thelauncher of a package uninstallation event. A user may interact with theicon grid application 260 by choosing icons to launch, or extractingicons to place into the landscape.

The Lifecycle Service may be a centralized service that manages theprocess of starting, stopping, putting to sleep, and waking upapplications. The Lifecycle Service also knows when applicationsterminate unexpectedly (crash). When any of these events happen, theservice's listeners are notified, and the Universe is one of thelisteners. The Universe accesses this service to start, stop, sleep andwake applications. In some embodiments, the Lifecycle Services provideapplication programming interfaces (APIs) for controlling the lifecycleof application processes running in the mixed reality system. TheLifecycle Services may spawn new processes to run application binarieswith a set of permissions, and call APIs on a predefined interfaceimplemented by the applications to control their lifecycle. TheLifecycle Service also provides a listener interface through which othermodules may keep track of applications beingstarted/stopped/paused/resumed. The Lifecycle Services may be a separateprogram from the launcher or the Universe. In some embodiments, theLifecycle Services may be a middlew are.

In some embodiments, as shown in FIG. 2 , the icon grid application 260comprises a centralized rendering service client 250 b and a ClientPrism 215 c. As discussed above, in some embodiments, applications thatdisplay content within a user's landscape may send their content to theUniverse via the centralized rendering service client 250 incommunication with the universe server 205. Here, the icon gridapplication 260, which provides the icons of installed applications onthe mixed reality system, for the launcher menu, is like any otherapplication that provides content for display in the user's landscape.However, in some embodiments, the icons within the icon gridapplication, when selected by a user, may instruct the Universe tolaunch and startup a new application, at which point, the newapplication may request the Universe to create a new Prism (e.g. throughUniverse Server 205) so that the application may provide content to bedisplayed into the new Prism. If the application is already executing,the Universe may request the application to open a new Prism.

The status bar application 270 comprises status indicators for the mixedreality system. The status indicators and the status bar application 270may not be adjustable by the user. The status indicators may beinitially populated by querying a first service for operating andmaintaining WiFi service, a second service for maintaining BluetoothService, and a third service for Status. When these services notify theStatus Bar application 270 of an updated status, the status bar mayadjust accordingly. The status bar provides the user quick glanceableinformation that they may react to quickly and efficiently from anywherein the system. In some embodiments, the status bar may be displayedabove the launcher. The four major sections in the status bar may be (1)global search, (2) notifications, (3) quick settings, and (4) power.Additional temporary sections may be added to the status bar when neededsuch as Music, Call, Sharing, etc.

When the user is in the Launcher menu, the status bar is condensed toglanceable icons. When the user swipes up to the top, it may trigger ananimation and the status bar may expand. The status bar may stay upabove the launcher while the user may swipe left and right through thelauncher panels. When the status bar is highlighted, it may expand andanimate forward. The sub-selection highlight may appear on the left bydefault, for example, on the global search. If there are other sectionsthat have more pressing content (e.g., recent notifications, lowbattery, etc.) the sub-selection highlight may appear on that sectioninstead.

The social panel application 280 may be composed of a series of contactsthat the user may interact with. The social panel may be initiallypopulated with a call to a Contacts Service for available contacts. Eachcontact may be added to the social panel and displayed to the user as anicon. When the social panel application 280 receives a new contact,updated contact, and removed contact events, the social panelapplication 280 may adjust its contacts information accordingly. Theuser may interact with contact icons by clicking on a contact icon topop up an option menu with the various contact providers available. Whenthe user selects a provider, the launcher application may start anassociated application with the contact's information.

The store panel application 290 may allow the user to search for,download, and install application(s) 140 for the mixed reality system.When a user requests to download and install an application, thelauncher application 230 may verify the user's identity with an identityverifying service (not shown), then may install the application with thePackage Manager. The Lifecycle Service may be invoked if the user startsthe application from the panel. In some embodiments, each panel in thelauncher may function as separate applications instead of as onelauncher application.

In some embodiments, the universe client 225 renders content specificfor the Universe. The universe server 205 does not render 3^(rd) partyapplications. This is because content within a Prism can only berendered by the universe client 225 and not the universe server 205.Thus, to render the infinity Prism, loader volume/Prism, and/orSecondary UI Prisms, work may need to be delegated to the universeclient 225 to render those particular types of content for the server.An infinity Prism may be used by the universe to render additionalgraphics around Prisms, for example, when two Prisms collide. InfinityPrisms are discussed further below. With the loader Prism and theSecondary UI Prisms, there may be specific communication between theuniverse server 205 and the universe client 225 to coordinate certainfunctionalities. For example, the universe server 205 may be told thatan application is done loading. The universe server 205 may then notifya client-side loader Prism that was currently loading the application.The loader Prism would have to react to the event that the applicationis done loading by showing the animation. Once the client-side loaderPrism is done showing the animation, the loader Prism may notify theuniverse server 205 that it is done animating. Then, the universe server205 may react to notification that the loader Prism is done animating byforce-placing the loader Prism, destroying the loader Prism, anddisplaying the App Prism with the rendered animation in place of theloader Prism). What has been disclosed is just one example of how theuniverse client 225 functions. One of ordinary skill in the art mayappreciate there may be other examples of when the universe client 225may assist the universe 130.

FIG. 3 shows an example bounding volume/Prism, according to someembodiments. Application content is presented to a user inside of one ormore bounding volumes called Prisms. As discussed above, when anon-immersive application is executed in the mixed reality system, itscontent is rendered inside of a Prism. The properties andcharacteristics of a Prism allow the Universe to consistently managePrisms within the user's landscape.

The volume space of a Prism 113 may have clear and definitive boundariesas indicated with dashed lines in FIG. 3 . The boundaries provide abounding volume for the virtual content 115 to only be displayed withinthe boundaries of the Prism 113. The boundaries of the Prism prevent thecontent from the application, displayed within the Prism, to overflow orspill outside of the Prism and into the user's landscape. The boundariesof the Prism 113 may not be displayed to the user when the user sees thevirtual content 115 displayed within the Prism 113. This is an importantfeature because, in order to maintain a realistic display of a 3Dcontent within the user's landscape, it is important to not show theboundaries of the Prism that are bounding the virtual content 115. Oneof ordinary skill appreciates the importance of not displaying theboundaries of the Prism that wraps around the virtual content 115 so thevirtual content may be displayed in a more realistic way in the user'slandscape. In contrast to 2D windows, the borders and boundaries of a 2Dwindow is generally displayed so the user of the computer displaying the2D windows may clearly distinguish content within one 2D window fromcontent from another 2D window. In some embodiments, however, it may beadvantageous to at least temporarily display the boundaries of thePrism, for example, to help troubleshoot problems with one or moreapplications.

Applications are given instances of Prisms 113 by the Universe to placecontent within. Applications may render 2D and/or 3D content within thePrism 113 using relative placement algorithms and/or arbitrarytransforms, but the Universe is still ultimately in charge of grossinteraction patterns such as content extraction. Multiple applicationsmay render to the Universe via the Prisms 113, with process boundariesseparating the Prisms.

Each Prism allocated in the Universe has an associated set of key-valueproperties that may be adjusted and may determine various bits ofbehavior or convey information about why a given Prism exists. Someproperties are read-only for normal applications, but for applicationswith the private API, these properties are writeable. A Prism 113 maycomprise Prism properties 310, application specific properties 320, andvirtual content 115. Additionally, some Prisms 113 comprise Secondary UIvolume 330 for providing users with additional Prism management options.However, in some embodiments, Prisms may not have a Secondary UI volume330, for example, because these other types of Prisms (e.g., LauncherMenu Prisms) may not require the features provided by the Secondary UIvolume 330. As with the boundaries of the Prisms, the Secondary UIvolume 330 may not be displayed to the user as well. When a user wantsto make changes to a Prism, the user may initiate a request to displayan Application Options Menu that displays the UI controls of the Prismwithin the volume space of the Secondary UI volume.

Depending on the application that they hold, Prisms may requiredifferent properties in order to afford the proper feedback and behaviorfor their content. Application developers may select from a number ofpre-programmed options for their Prism when they create theirapplication so their content may be represented correctly, based ontheir preferences. Below are examples of some of these options.

The Prism properties 310 define a Prism, at least in part, and allow theUniverse to manage and maintain the Prisms within the user's landscape.For example, Prism properties 310 may include one or more of a defaultsize, a maximum size, a minimum size, an anchor/placement type (e.g.,Option to billboard, etc.), a behavior of a given Prism to the anchortype, an anchor location, a child Prism spawn location, a child headpose highlight, an on surface behavior, an independent transformationcontrol, a resize vs. rescale indicator, an idle state timeout variable,etc. The Prism properties 310 allow the Universe the ability to trackand manage each and every Prism within the user's landscape. Having asingle application managing the virtual content displayed within theuser's landscape assures content displayed within a user's landscape aredisplayed in a consistent and reliable manner. Some of the Prismproperties 310 are further disclosed below.

Maximum, Minimum and Default Size: Applications may have upper and lowerbounds specified by an application developer (optionally, withadditional limits from the Universe). Additionally, applicationdevelopers may have a Default size when the application first launches.

Option to Billboard During Movement Sequence: Certain objects (e.g.content that is planar), make sense to billboard towards the user duringa movement sequence to encourage legibility and less management. Forexample, a certain content displayed on a planar surface may bepositioned at a specific location and/or relative to an object, buttheir orientation is automatically computed so that the contentdisplayed on the planar surface always faces the direction of the userviewing the content displayed on the planar surface. Other optional bodydynamics behaviors could be added to this as well.

Child Prism spawn location: Prisms may spawn children to create flexiblelayouts. The application developers should be able to determine aresponsive range of locations in which the children may spawn relativeto the parent Prism.

Child Head pose Highlight: Applications may be able to choose whetherhead pose highlight on children Prisms may be treated as separatehighlights or if it continues to highlight all Child/Parent Prisms asone unit.

Child Prism relational behavior: Prisms may determine whether theirchild Prism(s) may be anchored to them or not in translation, rotationand scale, and also choose whether the child Prism(s) will close withthe main Prism.

On Surface behavior: Prisms may be snapped to a surface and query thatsurface to determine if they want a size/scale change. If the surfacehas space, the Prism may resize to fit all or a percentage of thesurface and factor in field of view (FOV) of the user.

Independent transformation control: An application may requestindependent control over its translation, rotation, and scaling. Thismay allow the application to move and transform itself.

Resize vs. Scale: Some applications may choose to resize their boundsinstead of only scaling their content. This may accommodate more contentto be displayed within their bounds. This may function more likeexisting computer 2D windows.

Idle State Timeout: Applications may be able to choose how long it takesfor them to go into their idle state. This may handle situations whereapplications may wish to continue playing content even though they areout of view. For example, an application that displays live video maywish to continue to display content and play audio even though the userhas temporarily looked away.

The application specific properties 320 may be a list of key value pairsthat stores the application specific state information for each Prism.The list of key value pairs are specific to the application and the keyvalue pairs provide the state information of the content of theapplication that is being displayed or rendered within the Prism. Thelist of key value pairs may be different for each Prism, depending onthe application that is rendering into the Prism. For example, if theapplication is a video streaming application, some key value pairs mayinclude a video name, a viewed up to time for the video, an aspect ratiofor displaying the video, etc.

Both the Prism properties 310 and the application specific properties320 for each Prism may be stored within a data structure of the localdatabase 137. The Prism data are constantly updated while the user isoperating the mixed reality system and interacting with the Prisms. Asdiscussed above, the Prism instance data of the local database 137 maybe persisted by synchronizing with the external database 150 on aperiodic basis. In some embodiments, the local database 137 and theexternal database 150 may be synchronized in near real-time.

When a user launches an application in the Universe, the user may pull aPrism out of the Launcher Menu and place the resulting volume intospace. Other methods of launching an application may be used, such asclicking on an application icon. In some embodiments, the user may movethe Prism around in altitude and azimuth (on a spherical shell) bymoving a controller/input device (e.g., a totem) up or down. To move theobject closer or farther away, the user may use a joystick on the totemto “push” or “pull” the Prism, or may slide the user's finger over atouch sensitive part of the totem. This has the effect of changing theradius of the shell. In some embodiments, two buttons on the totem maycause the Prism to grow or shrink. Finally, rotating the totem itselfmay rotate the Prism. This assumes totems may have six degrees offreedom (DOF). This is consistent with the kind of controls used in VRpainting applications, for example, but the totem could be any suitableuser input device.

In some embodiments, Prisms may not allow themselves to be placed insuch a way that they fully or partially intersect other Prisms. Prismsmay either not intersect at all, or may not inhabit/be activelydisplaying at the exact same location (anchor point), with the exceptionthat Prisms may overlap a small amount for physics purposes, asdiscussed below. If more than one Prism is placed in the exact samelocation, the active application may be displayed and other applicationsanchored at the exact same location may be hidden. The user may be ableto tell there are multiple applications at a location by, for example,dots displayed in the volume. For example, if there are threePrisms/applications at a particular spot, there may be three dots. Ifthe user is viewing application #2 of three, then the second dot may bebrightened, while the other dots may be dimmed. The user may then swipeor scroll through different applications. The graphics may switch, andthe dots may update (e.g., by brightening the active dot) to show whichapplication is currently active.

In some embodiments, several Prisms may be co-located at the same anchorlocation. At first glance, this may seem like an odd thing to do. Withall of 3D space available for placing applications in the user'slandscape, why place them in the same spot? Actually, it makes perfectsense. For example, a user's favorite place to play virtual board gamesmay be on a kitchen table. In the morning the user may like to play“Ticket To Ride” while eating breakfast. But when the user gets homefrom work, the user may like to play “Risk” against a computer. The usermay have a plurality of board games located in the same spot, and switchbetween them when necessary.

In some embodiments, Prisms may be placed at an arbitrary location inspace. In this case, the Prism may be anchored by a center point of thecubic/rectangular volume. But if (e.g. during placement) a Prism ismoved near a horizontal surface in the landscape, the Prism may try tosnap to the surface. The anchor point may then become the center of thebottom plane of the Prism. Similarly, if a Prism is moved towards avertical surface (e.g. a wall) then it may try to snap to it, and theanchor point may become the side of the Prism that is next to thevertical surface.

The purpose of an anchor point may be to place the Prism so that it doesnot interpenetrate with the surface the Prism is anchored to. The anchorpoint may also move with the object it is anchored to. When multiplePrisms share the same location, that location may be the anchor pointand not the center point of their respective volumes. Applications don'tknow and don't need to know where they are located, but the applicationsmay ask their respective Prism to see how the respective Prism is beinganchored. Applications may also specify which anchoring types are valid.For example, it doesn't make sense to anchor a Halcyon to a verticalsurface.

All of the content (graphics) for the application may be containedwithin the volume of the Prism. The Universe may mask out graphics thatextend outside the Prism automatically. Because applications don't knowabout other applications in the world, the Universe may manageinteractions that happen between different Prisms of differentapplications.

The user interface design for placing Prisms may call for Prisms to swayin a physical way (like an object on a string) while the Prisms arebeing moved in the placement state. Instead of trying to predict whatkinds of physical behaviors different applications are going to want,the Prism may feed movement information to the application (through abinder interface) while it's being placed. The application may thenbehave appropriately.

There may also be physical behavior between Prisms as they are beingplaced. This may override the application's physicality implementation,and the application may stop receiving movement data.

Prisms may initially resist intersecting. If the user continues to pushtwo Prisms into the same location, then the Prisms may snap to theanchor location of the Prism it's intersecting with. This could be donein a way that feels elastic (e.g., similar to soap bubbles interactingwith one another) and is roughly based in physics.

Audio emitters may be placed as child nodes in an application's scenegraph. These nodes may be local to a root node transform. Thus, a Prismmay be moved wherein the movement of the Prism does not require theapplication to update the audio node's transform. The Universe may beresponsible for the final transform of the audio emitter to the worldspace. The Prism may also be responsible for constraining audio nodes toits boundaries. Applications may not emit audio from a point outside oftheir respective Prisms.

In some embodiments, it may not be desirable to spatialize audio. Forexample, if a user places a virtual TV on a wall, and is focused on theTV image, the TV's audio may be provided through to the user withoutmodification. This is likely to provide a better audio experience to theuser. In the case of surround sound, the audio signal already hasspatial information. The sound may be emitted from virtual speakersplaced in optimal locations relative to the TV.

In some embodiments, on a button press to control audio strength by theuser, the Universe may check the head pose to determine which Prism theuser is looking at and send a volume-up or volume-down event to thecorresponding Prism. The Prism may forward that information on to theapplication running in the Prism, and the application could decide howto interpret it. If there are no applications in focus in the landscape,then volume button settings may adjust the global volume.

In some embodiments, one difference between traditional 2D windows andPrisms 113 is that with 2D windows, borders that set the boundaries of a2D window are intended to be seen by users to provide a concrete borderfor encompassing content within the 2D window separate from contentoutside of the borders of the 2D window. However, in some embodiments,borders of the 3D windows (e.g., Prisms 113) are meant to be invisible.If users can see the outline (e.g., borders) of every Prism, it wouldbreak the illusion of “reality” and the virtual content displayed withinthe Prism having its borders displayed would appear likecomputing/digital/virtual content instead of real. In some embodiments,the borders may be displayed, for example to enable user manipulation asneeded.

Another difference is that 2D windows are commonly meant to becontrolled and/or interacted with by the user. For example, a closebutton may be always appearing in the upper right-hand corner of atraditional 2D window, or a menu bar may be displayed at the top borderof a 2D window. However, with the Prisms, a user generally does notinteract with the Prism and its boundaries. Instead, a secondary menu(e.g., an apps option menu) may be pulled down temporarily for the userto control and manage/manipulate the Prism from a list of options.

Furthermore, 2D windows are independent from its surroundings. Forexample, what is displayed on a computer screen does not automaticallychange if the user moves the screen. However, Prisms need to be placedin context with the real world. For example, each Prism may be placedinto the real world relative to (1) objects in the real environment suchas a wall, a table, etc.; (2) virtual objects created to provide abackdrop or canvas for the Prism to anchor to; and/or (3) the user. Insome embodiments, the Prisms may be placed in context with a passableworld as well as the real world.

Yet even further, in some embodiments, Prisms may not be allowed tooverlap/interpenetrate with one another, with the exception that Prismsmay overlap a small amount for physics purposes. For example, in someembodiments, when virtual content within two or more Prisms collide, thevirtual content may appear to show a bounce between the two virtualcontents when they appear to collide with one another. Here, the Prismsmay overlap for a small amount to create the effect of the bouncebetween the two virtual content. In some embodiments, when the boundingboxes for two or more Prisms collide, the Prism, and hence the Prism'scontent, may appear to bounce. However, 2D windows on a computer dooverlap and, in many cases, 2D windows may be cascaded on top of oneanother, hiding each other from view of the user. In some embodiments,if two Prisms are anchored at the same location in the user's landscape110, one of the Prisms may be displayed while the other Prism isminimized from display wherein an icon or a text or an image (or anyother visual indicator) is displayed to indicate to the user thatanother Prism is anchored at the exact same location. In someembodiments, an infinity Prism may be implemented to render additionalgraphics around Prisms, for example, when they collide. In someembodiments, an infinity Prism may be a Prism with its bounds set toinfinity. For example, if two Prisms are close to colliding, theUniverse may render a glow in the region of space between the twoPrisms. In order to handle these exceptions, the Universe may create aninfinity Prism that may encompass all space around/surrounding the twoPrisms, the user's entire field of view (what the user can currentlysee), the user's entire field of regard (what the user could see if theymoved around), etc. This may allow the Universe to draw graphicsanywhere between the two Prisms. In some embodiments, the infinity Prismmay not collide or interact in any way. In some embodiments, theinfinity Prism does not have a secondary UI, etc. In some embodiments,only the Universe may have access to the infinity Prism. The infinityPrism may be created at Universe initialization and may always bepresent until the Universe shuts down. In a second example, an infinityPrism may be useful in order to have a character (e.g. avatar, personalassistant, butterfly, animal, etc.) move between the other landscapeapps to, for example, explain to the user what each application isand/or how to use the application.

FIG. 4 shows an example launcher menu as seen by a user wearing, forexample, a head-mounted system 160 for launching applications to displayvirtual content, according to some embodiments. The launcher menu mayexist as its own application because of the elevated privileges itrequires to launch other applications. The launcher menu 400 maycomprise a status bar 410 and may have body-centric bounded volumes(e.g., Prisms) called panels. In some embodiments, the launcher panelcomprises several different applications that give the appearance of asingle application. For example, each panel may be a separateapplication, but may give the appearance of a single launcherapplication. The Universe may make explicit creation calls for eachpanel during a launcher menu summoning (e.g. a request to open thelauncher menu).

In some embodiments, a separate application process may render into eachpanel. The processes may use the same content rendering techniques asthe rest of the universe but, in some embodiments, with a one or more ofthe following exceptions: (1) The launcher menu may trigger the start ofother applications; (2) The launcher menu icons and widget icons may beextracted and placed in the landscape—placed icons may be referred to asshortcuts; (3) The launcher menu itself may be placed in the landscapeand moved around; (4) When content extraction has started on a launchericon, the launcher may load a placeholder content anchored to the user'shand, 6dof controller, or other controller such that once the extractionis finalized, the Universe launches the application into a Prism thatreplaces the placeholder content; and (5) the user may swipe betweenpanels in the launcher menu and expand individual panels. In someembodiments, the panels may not be implemented as carousels. Instead,the launcher panels may be co-located such that the main panel is alwaysfixed as the central panel and content from the minimized panels areanimated in and out of the central panel as the user performs a swipeleft, right, up, or down. In some embodiments, the launcher applicationitself may be placed within the user's passable world by the user,thereby creating a launcher shortcut or placed launcher within theuser's passable world and/or landscape.

In some embodiments, the launcher menu 400 may organize the series ofbody-centric panels as a panel carousel. The panel carousel may displayone panel at a time such that each panel is running its own applicationprocess, and/or each panel is a separate application. In someembodiments, only one current panel 405 may be expanded and visible at atime. Two body-centric placeholder panels (e.g., 440 a and 440 b) may bea fixed distance from the current expanded panel 405 on either side ofthe current expanded panel 405. The placeholder panels 440 a and 440 bmay be individual Prisms created by the Universe representing thecondensed form of the two panels adjacent to the current expanded panel405. In some embodiment, when the user swipes to the next panel, thepanel represented by the placeholder may become the current Panel. Theplaceholder panel's physical model may remain the same no matter whichpanel it is representing. In some embodiments, the only visualdifference between placeholders representing different panels is thelabel (e.g., Recent and Social as shown in FIG. 4 ). Other embodimentsmay have different panels in the same location but with a different iconspecific to the panel, see FIG. 11 .

In some embodiments, the launcher menu 400 manages the flow of panels,ideally caching the left and right panels. The panel carousel maycomprise a recently accessed applications panel 420, a generalapplications panel 425, a social application panel 430, and a storepanel (not shown). The recently accessed applications panel 420 maycomprise icons of applications recently accessed by the user. Therecently accessed applications panel 420 may be shown in a minimizedstate when the current panel 405 for the general applications panel 425is currently displayed. The recently accessed applications panel 420 maybe expanded to show its icons when the user swipes in the correspondingdirection (e.g., to the right) to bring the recently accessedapplications panel 420 front and center, or according to otherembodiments, contents pertaining to the respective panels may animate inand out of the central panel. In some embodiments, the recently accessedapplications panel 420 may display first, and may comprise a ring ofapplications. Any number of panels may be used, and those panels maydisplay in any order. The order and number of panels may be based onuser testing, ease of user, and/or user preference.

In some embodiments, the social applications panel 430 may be shown in aminimized state when the current panel 405 for the general applications425 (or any other panel) is currently displayed. The social applicationspanel 430 may be expanded to show its icons if the user swipes in thecorresponding direction (e.g., to the left) to bring the socialapplications panel 430 front and center, or in some embodiments,contents associated to the social applications are animated/displayedinto the central panel while the labels on the placeholder panels arechanged accordingly.

In some embodiments, the icons 450 of the general applications panel 425may represent icons within an icon grid application similar to the icongrid application 260 from FIG. 2 . The icons 450 may representapplications that are installed in the mixed reality system. A user maylaunch an application by selecting a particular icon 450. In someembodiments, a user may extract the icon 450 from the panel and placethe icon 450 in the user's landscape, at which point, the Universe mayinitiate the startup process for the application corresponding to theextracted icon 450. When the corresponding application completes itsrendering of its content, the extracted icon 450 may be replaced with aPrism, with the application's rendered content displayed within thePrism.

As discussed above, in some embodiments, the status bar 410 may comprisestatus indicators of the mixed reality system. The status indicators maybe initially populated by querying, for example, a first service foroperating and maintaining WiFi service, a second service for maintainingBluetooth Service, and a third service for Status. When these servicesnotify the Status Bar of an updated status, the status bar may adjustaccordingly. The status bar may contain status indicators for servicessuch as audio volume, current time, WiFi connection/strength, Bluetoothand power. Other services may be included in the status indicators, incombination with or in replacement of the services mentioned above.

In some embodiments, on a request to launch an application from thelauncher menu 400, an icon grid application may communicate with theUniverse, through for example, a Prism service and/or a Universeservice, to start an application. The universe may create a Prism with aspecified package name and/or component name. The Universe may give thePrism to the application using the application's listener. The Prism maynow be in “placement mode.” Once the user has placed the Prism, the usermay modify the Prism in any number of ways, for example, using theSecondary UI application options menu.

FIG. 5A-5B shows an example panel carousel change, according to someembodiments. FIG. 5A shows an example view of a launcher menu havingplaceholder panels represented as the Left 520 (e.g., 440 a from FIG. 4) and Right 530 (e.g., 440 b from FIG. 4 ) Panels before a Left to Rightswipe. Current active panel 505 shows an example icon grid depictingicons within the icon grid as “BB” within the active panel 505. In someembodiments, the icon grid may be a grid, a list, one or more columnswith one or more rows, a circle of icons, and/or one or more iconsforming any other shape. Miniature panel 535 shows the available panelsin the panel carousel (in some embodiments, the miniature panel 535 isnot shown to the user). Panel position indicators 540 showing a centerpanel represented as “C” pointing to the “B” panel, a left panelrepresented as “L” pointing to the “A” panel, and a right panelrepresented as “R” pointing to the “C” panel with the “D” panel notcurrently in view, and thus, not designated with a panel positionindicator 540 (in some embodiments, also not shown to the user).

FIG. 5B shows an example view of the launcher menu as shown in FIG. 5Aafter a Left to Right swipe. The new active panel 555 is now the “A”panel, having “AA” as the icons within the icon grid of new active panel555. The two placeholder panels Left 570 panel shows the “D” panel andthe Right 580 panel shows the “B” panel from the previous active panel505 in FIG. 5A. Additionally, Miniature panel 585 now shows theavailable panels in the panel carousel, as panels A, B, C, D. However,the Panel position indicators 590 have changed accordingly now showingthe center panel represented as “C” is now pointing to the “A” panel,the left panel represented as “L” is now pointing to the “D” panel andthe right panel represented as “R” is now pointing to the “B” panel.

In some embodiments, the placeholder panels themselves may remainconstant, with only the label reflecting the changing underlying Panel.For example, in FIG. 4 , if a user performed a Left to Right swipe,similar to the Left to Right swipe of FIG. 5B, the new active panelwould be the “Recent” panel with the right placeholder panel beingdisplayed with a new panel title of “General” and the left placeholderpanel being displayed with a new panel title of “Store” (assuming onlythe panel carousel only has the four panels disclosed in FIG. 4 ). Theimage of the placeholder panels may stay the same, but the titles of theleft and right placeholder panels may change accordingly. The Panelshifts may be based on the switching actions of a user motion, forexample, a swipe left to right. One of ordinary skill in the art mayappreciate that any other methods of switching between panels may alsobe used.

FIG. 6 shows a flowchart for an approach for starting a mixed realitysystem after a previous use of the mixed reality system, according tosome embodiments. Startup process 600 shows a process for poweringup/starting a mixed reality system, after a user has previously used themixed reality system and powered off the mixed reality system. At 610,the mixed reality system identifies the location of the user andestablishes a head pose to determine the location and/or orientation ofthe user (e.g. the user's head). The head-mounted system 160 may includeGPS for determining, at least in part, a location of the user.

At 620, the Universe may access a passable world system (e.g., externaldatabase 150) and retrieve instances of Prisms previously displayed inthe current user's landscape. The passable world may return instancedata so the Universe may reconstruct the previous state of the Prisms inthe user's landscape by reconstructing the user's local database 137with the retrieved Prisms instance data. The passable world systempersistently stores information about the environment of the user, forexample, which Prisms were previously active in the user's landscape 110as well as the state of each Prism when the user previously shut downthe system. Since the user's Prism instance data that is stored in thelocal database 137 may be erased when the user shuts down the mixedreality system, upon restart of the mixed reality system, the user'sPrism instance data in the local database 137 may be reconstructed basedon the instance data retrieved from the passable world system (e.g.,external database 150 from FIG. 1 ).

In some embodiments, the instance data may include, one or more of thefollowing, for each Prism, (1) Prism properties, wherein the Prismproperties may comprise location, orientation, extent width, extentheight, extent depth, anchor types, and/or anchor positions; and (2)application specific properties comprising state information of thevirtual content previously rendered by respective applications. Forexample, if an application rendering into the Prism is a video streamingapplication, the application specific information may include the titleof the video, the point in time the video was played up to, and theaspect ratio of the video being rendered. The application specificinformation may be a list of key-value pairs that stores stateinformation for each application. The list of key-value pairs may bespecific to respective applications and the key-value pairs may providethe state information for the content of the application that is beingdisplayed or rendered within the Prism.

In some embodiments, each user 103 may have their own Prism instancedata stored in their local database 137 while a master Prism databaseinstance data is stored in the passable world system (e.g., externaldatabase 150) that includes other Prism instance data for other users ofthe mixed reality system and/or Prism instance data for other locationsbesides the user's current location. Prism data for users of the mixedreality system may be stored in the passable world system. In someembodiments, a first user may want to share a virtual content displayedwithin a Prism from the first user's landscape with a second user who isusing a mixed reality system at the same location, or in someembodiments, at a different physical location than the first user. Sincethe Prism information is available to the first user at the localdatabase of the first user, upon sharing the Prism to the second user,instance data for the Prism from the first user may be uploaded to thepassable world system. The instance data for the Prism may then betransmitted to, or retrieved by, the second user to be stored in a localdatabase of the second user so that a universe application of the seconduser may reconstruct the Prism's instance data into the second user'slocal database and consequently, display the Prism into the landscape ofthe second user.

At 630, the Prism may be restored at the current location of the user.Once the instance data for the Prisms that were previously deployed atthe user's location is reconstructed in the user's local database 137,the respective applications corresponding to the Prisms that werepreviously deployed at the user's current location may be launched.Since the mixed reality system is just starting up, the applicationswhich were previously providing content to the mixed reality system mayneed to be started, but at this point, only the applications that werepreviously providing content to the user's landscape need to belaunched. As part of the launching of the applications, the applicationsmay request that the Universe create Prisms for the applications torender content into, wherein the Prisms to be created correspond to thePrisms that were previously deployed prior the user shutting down themixed reality system, for displaying in the user's landscape. Once theapplications are launched and the Prisms are created, the applicationsmay provide the content to the Universe for rendering the content intorespective Prisms.

In some embodiments, one application may provide content to more thanone Prism deployed at the user's location. In such an embodiment, theone application may deploy multiple instances or processes of the oneapplication to render content for each instance or process of the oneapplication into one or more Prisms.

At 640, the Prisms may be displayed into the user's landscape at alocation that the Prisms were displayed prior to the user shutting downthe mixed reality system. Because the Prisms and the applications thatrender into the Prisms are persisted, the mixed reality system mayreconstruct the user's landscape by deploying/displaying the Prismsalong with the respective virtual content at the location of the user'slandscape based at least in part on each the Prism properties of each ofthe Prisms. In some embodiments, placeholder Prisms such as loadervolumes 210 from FIG. 2 may be displayed in a location of a currentlyloading Prism upon startup. In such embodiments, once the new Prisms arecreated and restored, displaying the new Prisms in place of theplaceholder Prisms may be as simple as replacing the respectiveplaceholder Prism with the newly created Prism at the location of theplaceholder Prism. If a Prism had the property marking it as beinganchored to the user instead of the landscape, then that Prism may berestored in a position relative to the user's current position.

In some embodiments, an application rendering in a Prism may be deployedin a state consistent with the state when the user previously shut downthe mixed reality system. For example, a user's field of view of theuser's landscape may have had four Prisms deployed where one of the fourPrisms had an application that was in a sleeping state, while the otherthree Prisms each had an application that was actively rendering. Whenthe mixed reality system is restarted and the four Prisms are displayed,three of the Prisms may have an actively rendering state while thefourth Prism may have an application that is in a sleeping state untileither the user reactivates the sleeping state of the fourth Prism, orthe Prism is configured such that if the user looks in its generalvicinity, the state may be automatically changed to actively rendering.Changing the state of a Prism is further disclosed below. In someembodiments, other methods may be used to change the state of a Prism.

At 650, the user's local database may be constantly (or at a regularinterval, or as triggered by an event) and automatically updated withPrism information as the user interacts with the Prisms in the user'slandscape. For example, when applications for certain Prisms changestate from actively rendering to active but not rendering, or tosleeping, the user's local database is updated with the updated stateinformation for the application within that Prism. The updatedinformation from the user's local database may be immediately orperiodically updating the passable world system (e.g., external database150) to ensure the passable world system is kept up to date with thechanging state of the applications and Prisms within the user'slandscape. One of ordinary skill in the art may appreciate that this ismerely one example of how a Prism property or an application specificproperty is changed and how updates are made to the local database andthen to the external database and that many other similar types ofchanges to a Prism or to the application that is providing content tothe Prism may be made.

The problem of keeping virtual content from one application displayed ina 3D spatial environment private from, but still interactable with,virtual content from a different application displayed in the 3D spatialenvironment may be solved by having one program manage the contentcreation (e.g., application(s) 140 from FIG. 1 ) and a separate program(e.g., universe application 130 from FIG. 1 ) manage the contentdisplay.

FIG. 7 shows a flowchart for an approach for displaying, by a universeapplication, a virtual content provided by an application into a mixedreality 3D spatial environment, according to some embodiments. At step710, a universe application of the user may receive a request to displayvirtual content in the user's landscape (e.g., 3D spatial environment).In some embodiments, the request may have been initiated by a userinteraction with a launcher menu wherein the user chooses an icon froman icon grid to launch an application to provide virtual content intothe user's landscape. As discussed above, one application may (e.g.,application 140 from FIG. 1 ) manage the content creation and a separateapplication (e.g., universe application 130 from FIG. 1 ) may manage thecontent display. This separation of processing may allow multipleapplications to create content to display within the user's landscapewhile the management of the display of the content is managed by aseparate application to ensure the multiple virtual content displayedwithin the user's landscape is displayed accurately and realistically tothe user, especially when the virtual content begins to interact withone another.

At 720, the Universe may create a Prism so that the content created bythe application may be displayed within the boundaries of the Prism inthe user's landscape. It is important to note that although the Prismmay include boundaries which defines the boundaries of the volumetricspace of the Prism, the borders of the boundaries of the Prism may notbe displayed to the user. This may be important because the virtualcontent rendered inside the Prism to be displayed in the user'slandscape should appear as real as possible to the user viewing thevirtual content in the user's landscape. If the borders of theboundaries volumetric space bounding the virtual content is viewable tothe user, the virtual content may inevitably appear as computergenerated within a confined space. Therefore, although the Prism mayinclude borders and boundaries to confine the virtual content within theboundaries of the Prism, the borders may not be displayed to the user.Borders of the Prisms may be displayed in certain circumstances such asduring system development and testing, etc. In some embodiments, theborders/boundaries of the Prism may be displayed to the user, forexample, during placement so the user can more easily understand wherethe Prism will be located.

In some embodiments a single application may be requested by the user todisplay multiple virtual content within the user's landscape. Eachvirtual content displayed by the single application may be associated toand displayed by its own individual Prism, or may share a single Prism.Therefore, in some embodiments, if the application requests additionalvirtual content to be displayed, the Universe may create one or moreadditional Prisms for the application, where each of the additionalcontent may require its own Prism to be deployed/anchored within theuser's landscape.

The Prism created may be provided with a unique identifier of theapplication requesting to display the content so that the when theapplication provides the content to be displayed in the user'slandscape, the Universe may be able to uniquely identify the Prism torender the content received from the application. Furthermore, the Prismmay be created with automatically having a set of functionalitiespredefined, which, in some embodiments, may be predefined by theapplication and/or the Universe. For example, the set of functionalitiesmay include a default min/max size allowed for the Prism, an aspectratio for resizing the Prism based on how far or close the Prism may beanchored in the user's landscape relative to the user, an anchor typefor anchoring the Prism in the user's landscape, etc. The set offunctionalities may be stored in the local database as Prism properties.Furthermore, depending on the application invoking the creation of thePrism, certain application specific properties may also be created forthe Prism as well.

At 730, the universe may receive the virtual content created by theapplication. Upon receipt of the content, the Universe may render thevirtual content to be displayed within the boundaries of the Prism. Asdiscussed above in FIG. 2 , in some embodiments, the application maycreate the content and may send the content via the centralizedrendering service client to the universe centralized rendering servicerunning in the Universe. From there, the Universe may render the contentinto the Prism created for the application requesting to display thecontent. The Universe may also be responsible for ensuring that thePrism is displayed correctly in the user's landscape. For example, theUniverse may make sure that the Prism does not overlap with other Prismsin the user's landscape. The Universe makes sure that a new Prism cannotbe placed in an overlapping position relative to existing Prisms or realworld content, unless an exception as described above applies (e.g.,common anchor point). The Universe may not allow a user to place a Prismin a location where Prisms overlap. In some embodiments, theclient/server communication with the applications may not be necessary.If the operating system (e.g. the portion of the system that manages therendering) is configured for multi-threading, then the applicationscould render themselves into Prisms directly without the client/serverstructure. A central authority, like the Universe server could then beresponsible for managing collisions, moving, inputs, etc.

At 740, the Prism may be associated to (1) a fixed location in space inthe user's landscape, (2) a physical object within the user's landscape,(3) a virtual object displayed within the user's landscape, or (4) afixed distance relative to a particular object (e.g., the user's body orbody part, also hereinafter referred to a body-centric anchor), based oneither a user's input or a default anchoring rule that may apply to theparticular application or Prism. In some embodiments, the body-centricanchor may be relative to the user's body parts (e.g., body fixed), or ahead/view position (e.g., head fixed). For example, the Prism may beassociated to a body fixed position such that the Prism is fixedrelative to the user's body. If the user moves his head, the content maynot move, but if the user walks, the Prism may move with the user tomaintain the fixed position relative to the user's body. As anotherexample, the Prism may be associated to a head fixed position such thatthe Prism is fixed relative to a user's head or pose. If the userrotates his head, the Prism may move relative to the user's headmovement. Also, if the user walks, the content may also move relative tothe user's head.

At 750, the Prism is placed/anchored into the user's landscape.Depending on how the user launched the application, the Prism mayalready have been associated with a particular anchor/location. Forexample, the user may have launched the application by dragging theapplication icon from an icon grid within a launcher menu and placed theicon on a particular location in the user's landscape, at which point,the Prism may be created at that location and the content is renderedinto the Prism at that particular location.

FIG. 8 shows an example of a diagram for managing application states,according to some embodiments. The problem of high computing load formanaging, rendering and displaying augmented reality content may besolved by automatically changing the state of Prisms based on a relativeposition and location of the Prisms to the user and in some embodiments,the user's field of view. Diagram 800 shows a bird's eye view of auser's location, according to some embodiments. The user 805 is shown tobe within a volumetric grid that is a coarse representation of theplanet (e.g., a physical environment) in an x, y, and z axis. Grid linesare shown for illustrative purposes to indicate a distance of the user805 to Prisms have different Prism states of 870. Here, a 2Drepresentation of the user's current location is depicted. One ofordinary skill in the art may appreciate this view may be a 3Drepresentation of the user's current location. However, for simplicitypurposes, FIG. 8 has been illustrated to show a 2D space.

The user 805 is shown to be in the center of a three by three cellexternal zone 810. Additionally, the external zone 810 is segmented intotwo circular areas comprising an active zone 820 and a buffer zone 830.The external zone 810 comprises a three by three cell grid wherein theuser 805 is located in the center of the grid. As the user 805 movesthrough the user's landscape, other Prisms associated with respectiveapplications may enter and leave the various zones as depicted in FIG. 8. The active zone 820 illustrates an area around the user 805 (e.g., acircular area) to indicate certain Prism states 870 of Prisms locatedwithin the active zone 820. For example, sleeping Prisms within theactive zone 820 of the user 805 need to startup, while other Prisms thatare running must remain running, even though their states are active butnot rendering 870 b located within the active zone 820.

Buffer zone 830 is a zone that is further away from the user than theactive zone 820. Prisms within this zone may not require a state change.For example, running Prisms remain running as in, for example, the onePrism located in the buffer zone having a Prism state of Active but notrendering 870 b. As another example, sleeping Prisms may remain sleepingas in the two Prisms within the buffer zone 830 with a Prism state ofsleeping 870 a. Prisms outside of the buffer zone 830 but within theexternal zone 810 may remain in a sleeping state since they are farenough from the user 805 that they may not need change state. Finally,Prisms located in the outside relevant cells 815 may have Prism statesof not loaded, sleeping or background because they are located so faraway from the user 805 that it may be beneficial to the mixed realitysystem to save processing resource by not running those particular Prismuntil the user 805 is within a closer distance from the respectivePrisms.

Prism state legend 860 discloses the various different Prism states ofPrisms based on a color scheme, wherein a red color Prism indicates aPrism state of sleeping 870 a, an orange color Prism indicates a Prismstate of Active, no render 870 b, a green color Prism indicates a Prismstate of Active and render 870 c, a purple color Prism indicates a Prismstate of not loaded 870 d, and a blue color Prism indicates a Prismstate of Background, active no render 870 e.

The user 805 may have a particular field of view 840 where Prisms withinthe field of view 840 may be in a frustum 850. Wherein a frustum is a 3Dregion that is visible on a screen of the user 805 (e.g., via ahead-mounted system 160 from FIG. 1 ). Running Prisms within the frustum850 should be rendering, as indicated by Prisms having Prism state ofActive and Rendering 870 c. Since there is one Prism with a Prism stateof sleeping 870 a located within the frustum 850 and buffer zone 830,that particular Prism may be starting up and rendering when thatparticular Prism is located within both the buffer zone 830 and thefrustum 850. In some embodiments, a head pose frustum may be used forculling the rendering of Prisms, wherein frustum culling is a method ofdetermining hidden surfaces, such as hidden surfaces of Prisms.

Table 1 below illustrates a Prism state change chart wherein rows of thefirst column describe a current state of a Prism. The rows of thesubsequent columns describe the change in Prism state from the currentstate into the new state as the Prism moves from a current zone into thenew zone as indicated by the subsequent columns. The values in the rowsof the subsequent columns describe the new state the Prism may changeinto (if at all), based on a relative distance of the Prism to the user805.

TABLE 1 Prism State Change Chart New Event Active Buffer Outside Zone +Zone + External relevant Current State Active Zone Frustum Buffer ZoneFrustum Zone Cells Sleeping Active Active + Sleeping Sleeping SleepingNot Loaded No Render Render Active + No Active + No Active + Active + NoActive + Sleeping Not Loaded Render Render Render Render Render Active +Active + No Active + Active + No Active + Sleeping Not Loaded RenderRender Render Render Render Not Loaded Active + No Active + SleepingSleeping Sleeping Not Loaded Render Render Background Active + NoActive + Active + No Active + Active + No Active + No Capable App RenderRender Render Render Render Render

For example, if a Prism is in a current state of “Sleeping” and based onthe user 805's movement that may place the Prism within the active zone820 relative to the user 805, the Prism state of the Prism may bechanged to “Active No Render”, because sleeping Prisms should start upwhen it is within the active zone 820 of the user 805. Additionally,following the same example, if a Prism is in a current state of“Sleeping” and once the Prism is within both the buffer zone 830 andfrustum 850, the state remains unchanged since within a buffer zone 830,sleeping apps may remain sleeping, even within the buffer zone 830 andfrustum 850. However, once that sleeping Prism transitions into theactive zone 820 and frustum 850, the Prism may begin to start up and thePrism state may change to Active and render 870 c.

FIG. 9 shows a flowchart for managing a Prism state relative to a userlocation, according to some embodiments. At 910, a spatial position ofPrisms may be recorded in a volumetric grid. The spatial position may bean anchoring location of a particular Prism that is placed/anchoredwithin a user's landscape. For example, a Prism may be anchored to anobject within the user's landscape, such as a wall or a table. Asanother example, a Prism may be anchored to a fixed location in spacethat is not associated to any particular object within the user'slandscape, such as a floating 3D globe of planet Earth. When a Prism isstarted, its spatial position is recorded in a volumetric grid that mayhave an x, y and z axis relative to the world to identify the locationof the Prism that the content of the Prism may be rendered into. EachPrism placed into the user's landscape may have a particular locationstored (e.g., as a volumetric grid) within the Prism's data structurestored in the passable world system. The volumetric grid, as discussedabove, may represent a coarse representation of the planet in an x, yand z axis format. Any other plausible coordinate or relative placementsystem could also be used.

At 920, a cell of the volumetric grid representing a 3D spatial area isidentified. The cell may include one or more Prisms if the Prisms arespatially in close vicinity to one another. In some embodiments, thewidth of the cell may be equal to or larger than the radius of an activezone. In some embodiments, the width of the cell may be less than theradius of an active zone. The radius of the active zone denotes ameasurement of distance from the user using a mixed reality device to apre-configured distance (e.g., 25 meters) that would define a circularand/or spherical (or other suitable shaped) area of space as an Activezone 820 as depicted in FIG. 8 . Alternatively, the volumetric grid maybe divided into cells having a predefined length to further establishboundaries and borders corresponding to distances from the user usingthe mixed reality device to establish the different zones to implementembodiments of this disclosure.

At 930, a distance of known positions of each Prism within the cell andneighboring cells is determined when a user using a mixed reality devicemoves into a location of the cell. When a user using the mixed realitydevice moves through the user's landscape, a distance check to eachPrisms' last known position is checked in order to manage the state ofthe Prisms within the vicinity of the user. In-order to preventperforming distance check across Prisms all over the world, the systemmay minimize the amount of distance checking/calculating/determining byusing the volumetric grid and restricting thesearching/checking/calculating/determining to only the cells around themixed reality devices (e.g., the cell and neighboring cells). In someembodiments, only Prisms in cells occupied by the user are distancechecked. In some embodiments, only Prisms in cells occupied by the useror adjacent to the cells occupied by the user are distance checked. Insome embodiments, Prisms in all of the cells within the user's FOV/viewfrustum are distance checked. In some embodiments, Prisms within adistance of a user may be distance checked, such as within 10 feet or 25feet of the user.

At 940, depending on the distance between the Prism and the user, thePrism's state may be modified. In addition to the distance, otherfactors may dictate whether or not a Prism's state may be modified suchas the current state of the Prism relative to the different zones andthe field of view of the user. For example, if a Prism's current stateis sleeping and the user moves to a location where the Prism is locatedwithin an active zone 820 and a frustum 850, then the Prism's state maybe changed from sleeping to active and rendering, per Table 1. However,if the sleeping Prism is located within a buffer zone 830 and a frustum850, then the Prism's state may not be changed because according toTable 1, a current state of sleeping remains in the sleeping state whenthe Prism is moved into the buffer zone and frustum zone. The bufferzone, as disclosed above, mainly prevents intermittent or rapid changesto the state of the Prism. The head pose of the user may modify thestate of a Prism because the head pose of the user helps to define theuser's field of view, which ultimately aids in the definition of thefrustum zone 850. As disclosed above, a head pose of the user is ameasurement of a location and/or orientation of the user's head. Thehead pose information can be used to render a scene to match a user'sdynamically changing head location and orientation to provide anincreased sense of immersion in a virtual/augmented/mixed space.

In some embodiments, a wall may separate a user from a Prism, such thatthe Prism may remain in a sleep state even though the distance of theuser places the Prism within an active zone 820 and/or a field of viewdirection of the user places the Prism within the frustum zone 850, thestate of the Prism may not be changed from sleeping because of theocclusion of the wall, which prevents the user from seeing the Prism.Further computational processing may be avoided by the systemrecognizing that even though the distance and head pose may indicate astate change for the Prism, other factors such as walls or objectsblocking the Prism may keep the Prism's state unchanged.

Managing Prism Relationships and Groupings

In some embodiments, the centralized rendering system may not havesupport for transforming/managing Client Prisms 215 as part of a larger(e.g. universe space) scene graph. A Universe Prism may be an instanceof a Client Prism with additional logic and functionality. In theUniverse, the capability to transform a group of Client Prisms 215 maybe handled via a transform tree. For example, each panel of the launchermay be a separate application, wherein each panel corresponds to its ownPrism. It may be beneficial for the Universe to be able to group themultiple Prisms associated with the launcher into a group so themultiple Prisms may be managed together such that when one Prism withinthe group is modified, the other Prisms within the group may be affectedand thus may need to be modified as well. This may enable multipleapplications to appear and behave as if it were a single application.For example, if the launcher is anchored to the user's body (e.g., bodydynamic), as the user moves throughout his environment, the location ofthe launcher, as well as the location of its panels corresponding tomultiple Prisms may also need to be modified/transformed as a group.

As another example, an application requesting a Prism may have asecondary UI (with a close button, pin/unpin, settings, etc.) that ispositioned relatively close to the application's Prism. The Universe maygroup these two Prisms together as a group and manage the two Prisms asa group having two nodes. Managing the association of multiple Prisms ina hierarchical/relational manner and/or in a logical grouping structuremay be handled with a transform tree and/or a group tree (discussedbelow).

In some embodiments, a transform tree is a tree data structureconfigured to transform/organize Prisms and/or groups of Prisms in theUniverse, and may be rendered by the central rendering system. Atransform tree structure may have a scene graph of nodes where each nodeprovides a moveable local space corresponding to a Prism. A node of thescene graph may correspond to a Prism. A Prism may be a tree node. Theremay be other tree nodes that are not Prisms. These “other tree nodes”may be used for grouping content or as an intermediate transform.Furthermore, nodes may be grouped with other nodes such that when onenode is transformed, the other nodes in the same group are transformedalong with it. In some embodiments, a group tree may manage the groupingof nodes.

A group tree may be a tree data structure that stores a logical groupingof nodes.

In some embodiments, since Prisms may not overlap and may be treated asindependent content displayed in the user's landscape, this mayintroduce the problem of virtual content from one application not beingable to closely interact with virtual content from a differentapplication. However, creating multiple smaller Prisms closely fitaround virtual content from a single application may solve this problem.For example, there may be multiple Prisms open for a single application,but the user may perceive the multiple Prisms as one unified system(e.g., parent/child Prisms of a transform tree). Additionally, thePrisms may also be grouped together such that some Prisms may be peersto one another essentially. If all of the content from one applicationis within a big box, there may be more “air” or dead space within thePrism. This may prevent content from different applications frominteracting with content in those areas, thus preventing certain typesof “realistic” interactions from occurring.

For example, an interaction may be a simple collision of two 3D cars.The collision may not seem “real” to a user if each of the Prisms thatthe two 3D cars are rendered into have so much “dead space” in betweenthe invisible boundaries of the Prisms relative to the 3D cars that auser may not be able to see the two cars actually contacting oneanother. However, if the Prisms are smaller and closely fit around eachof the 3D cars, from certain view angles, it may appear to the user asthough the two 3D cars are slightly touching one another and bouncingoff one another as a result of the collision.

FIG. 10 shows an example tree node of a scene graph, according to someembodiments. Tree node 1000 depicts an example tree node for thelauncher menu. The universe node 1002 is the root node in this tree nodeof a scene graph of the user's mixed reality system. As the mixedreality system starts up, a launcher application may be started todisplay to the user as a launcher menu. As the launcher application islaunched, a launcher tree node 1004 representing the launcher menu'sposition in the scene graph may be created. The launcher tree node 1004may serve as an anchor for other nodes that depend from or is a child ofthe launcher menu.

For example, an icon grid application may also be launched. An icon gridtree node 1009 a may be added to the tree node with a scene graphidentifier (SGID) of the launcher application. Since the icon gridapplication may be displayed as one panel of a carousel of panels forthe launcher menu, the icon grid tree nodes 1009 may be children nodesof the launcher tree node 1004, as depicted within the child nodes 1008indicated with the dash border labeled as Launcher Panes. More of theLauncher's subordinate applications may be launched and added to thetree node as child nodes (e.g., recently assessed applications tree node1027 and social application tree node 1029) of the launcher tree node1004. The group of nodes associated to the launcher menu may be groupedtogether as launcher group 1006.

When the launcher menu is moved (e.g., billboard), the nodes within thelauncher group 1006 may be moved as a group. Setting the launcher nodeas an anchor node may be necessary for providing an exact means toposition/billboard the launcher menu precisely, rather than applyinglogic to one of the launcher panels and hoping for the best to maintainthe groupings of the other related panels. The child nodes (e.g., theicon grid, the recently accessed applications of 102 and the socialapplications) are automatically repositioned relative to the launchertree node. The Universe may own the entire scene graph of the pluralityof Prisms. In some embodiments, each Prism may start off with a nullparent and zero children, effectively making it a scene graph with onenode. In some embodiments, when an application requests a Prism, theUniverse may create (1) a node for the requested Prism, and (2) a nodefor the Secondary UI. The Universe may then position the Secondary UIrelative to the Prism. In some embodiments, the Universe may place thesecondary UI node as a child of the Prism node.

Transform Tree

As discussed above, a transform tree is a tree data structure configuredto transform/organize Prisms and/or groups of Prisms in the Universe. Atransform tree structure may be considered a scene graph comprisingnodes, where each node provides a moveable local space that maycorrespond to a Prism. A node of the scene graph may correspond to aPrism or it may be a node used only for transforming a group of Prismsor other tree nodes. In this case, nodes may be grouped with other nodesunder a parent node such that when one node is transformed, the othernodes in the same group are transformed along with it. In someembodiments, Prisms are in a transform tree with just one Prism (i.e.itself), but when the need arises to parent one Prism with another, theability is there. In some embodiments, the Prism scenegraph/transformtree may differ from a conventional scene graph in that, when a childPrism changes its parent, the world transform of the child may notchange. For example, if a Prism is rendering in front of the user, whenthe Prism's parent changes, the Prism would not move from its currentposition. In some embodiments, the matrices of each Prism within thetransform tree may be column-major, which means calculating the worldmatrix of a Prism within the tree must be done by appending the parenttransform to the left of the Prism's local transform. For example, aPrism's world transform may equal the parent's world transform times thePrism's local transform.

In some embodiments, basic tree logic may be defined as one or more ofthe following: (1) A node may set itself as a child to another node; (2)A node may set a different node as a child to itself; (3) A node may getits parent; (4) A node may get its children; (5) A node may remove anyof its children (this doesn't orphan the children—they may just attachto this node's parent, or attach to any other node in the tree,including the root Universe node, which may put it in world space); and(6) A node may remove its parent. Inserting and removing nodes mayupdate the local transforms of the nodes and its children. This meanschanging the parent of a node may not change its world transform, infact the world transforms may be preserved. In some embodiments, anyoperation that changes the local transform of a node may update theworld transforms of the node and its children.

In some embodiments, each node in the transform tree may contain one ormore of the following matrix components: (1) Local transform; (2)Rotation matrix (this may be used to compute the local transform); (3)Scale vector (this may be used to compute the local transform); (4)Position vector (this may be used to compute the local transform); (5)Local to world transform (this may be cached for convenience); and (6)Local to world scale (this may be cached for convenience). In someembodiments, the local transform may be thought of as the transformapplied to its parent node. If the node does not have a parent, then thelocal transform may equal the local to world transform. The localtransform may be computed as follows:

local transform=position*rotation matrix*scale

This formula may be read as, “Transform this node, using the parent nodeas its origin, by first scaling it, then rotate it, then change itsposition relative to its origin node.” Rotation, scale, and position maybe kept separate to make it easy to provide getters/setters to modifyany of them separately through an interface. When a node changesposition, the child node world transforms may be updated with it. Whenthese matrices are multiplied in sequence, then a local-to-world matrixmay be constructed. For the sake of efficiency, the calculatedlocal-to-world matrix may also be stored. In some embodiments, tocalculate the local to world transform of a node, the following equationmay be used:

world transform=parent world transform*local transform

Note the order of matrix multiplications. In some embodiments, thematrices are column-major, implying that x, y, z and position are laidout in each column. This means parent node matrices always appear to theleft of the children matrices. FIG. 10B illustrates a small transformtree 1010 and how each world transform for each node is calculated.

Other suitable methods or systems utilizing transform trees may be used.

Group Tree

In some embodiments, an additional feature of the transform tree is theability to group any Prism within the transform tree with any otherPrism. The Prisms in the group may be in the same, or different,transform tree. Additionally, the Prisms can be a child or descendant ofanother Prism within the same group and Transform Tree. In someembodiments, when a transform is applied to one of the Prisms in thegroup, all of the other Prisms within the group are transformed alongwith it. The Prism group logic takes care of updating the transforms inwhatever transform trees the group members belong to. Any Prism notwithin the group, but that is a child of a Prism that is in the group,may also transform along with the group. When a Prism in the group istransformed, its children are also transformed. For example, if Prisms Aand B are in a group, and Prism C is a child of Prism B but not withinthe group, and Prism B is moved 1 meter to the left, then both Prisms Band C are moved one meter to the left. If, however, Prism C moves 1meter to the left, then only Prism C is moved 1 meter to the left. Thepresent disclosure shares some features with conventional scene graphs,but may have additional functionality/behavior as well, such as 1) theability of an indirect descendent of a node to be grouped with theindirect descendent so the node and indirect descendent undergo the sametransformation (if they are grouped), and/or 2) the ability to group twonodes that are located on entirely separate transform trees. In someembodiments, an indirect descendent may be a node that is not directlyconnected to the parent node, for example, possibly separated by two ormore intervening nodes. In some embodiments, the grouping logic may beimplemented by creating a Root Group Node for all Prisms that belong inthe current group. If any Prisms in the same group are descendants ofother Prisms within the group, then they are also descendants of thosePrisms within the transform group tree. Note that, in some embodiments,a separate data structure may not be created for a group; instead thegroup may have an additional flag attached to the Prism. In someembodiments, if any Prism within the group is transformed, a transformmay be calculated and applied to the Root Group Node. This transform maypropagate down the Transform Tree like any other transform, where thetransform propagates down to each child within the transform tree, aswell as each Prism within the group.

Following are various embodiments of adding and removing nodes to andfrom a group tree. FIGS. 10C-10E illustrate one embodiment for creatinga new group. For the transform tree 1020 in FIG. 10C, this embodimentwill illustrate a grouping of nodes A and E together, which arecurrently in separate transform trees. (1) A determination is made as towhether or not A or E is already in a group. Currently, neither A nor Ehave a group-parent, so neither are in a group. Therefore, create aGroup Root. At FIG. 10D, a group tree 1022 having a Group Root 1024 iscreated. (2) Assign A and E's group parent to the Group Root 1024 asdepicted in FIG. 10E. (3) Determine if E is an ancestor (or parent) ofA. A has no parents, so E is not an ancestor of A. (4) Determine if A isan ancestor (or parent) of E. E has no parents, so A is not an ancestorof E.

FIGS. 10E-10I illustrate another embodiment for creating a new group.For the transform tree 1030 in FIG. 10F, this embodiment illustrates agrouping of nodes A and C together, where node C is a child node of nodeA as depicted in the transform tree 1030. (1) A determination is made asto whether A or C is already in a group. Currently, neither A nor C havea group-parent, so neither are in a group. Therefore, create a GroupRoot. At FIG. 10G, a group tree 1032 having a Group Root 1034 iscreated. (2) A and C's group parent is the Group Root 1034. Assign A andC to Group Root 1034 as depicted in FIG. 10H. (3) Determine if C is anancestor (or parent) of A. A has no parents, so C is not an ancestor ofA. (4) Determine if A is an ancestor (or parent) of E. A is an ancestorof C. Therefore, re-assign C's group parent to A.

FIGS. 10J-10K illustrate an embodiment of adding an ungrouped node to anexisting group. For the transform tree 1040 and group tree 1042 in FIG.10J, this embodiment illustrates a grouping of nodes C and E from thetransform tree 1040 into a same group. (1) Determine if C or E isalready in a group. As depicted in FIG. 10J, E is in a group with A. (2)Determine C's group parent—find a parent or ancestor of C that belongsin the same group as E. Here, A is in a group with E. A is an ancestorof C. Assign C's group parent to A (See FIG. 10K). (3) Determine if Chas an ancestor or parent of any other nodes in the group. C has nochildren. Processing stops.

FIGS. 10L-10N illustrate another embodiment of adding an ungrouped nodeto an existing group. For the transform tree 1050 and group tree 1052 inFIG. 10L, this embodiment illustrates a grouping of nodes B and E fromthe transform tree 1050 into the same group (e.g., group tree 1052). (1)Determine if B or E is already in a group. Here, E is in a group with Aand C. (2) Determine B's group parent by finding a parent or ancestor ofB that belongs in the same group as E. Here, A is in a group with E andis a parent of B. Assign B's group parent to A (see FIG. (3) Determineif B is a parent or ancestor of any of the other nodes in the group.Here, C is a child or descendent of B. Change C's group parent to B (seeFIG. 10N).

FIGS. 10O-10R illustrate another embodiment of adding an ungrouped nodeto an existing group. For the transform tree 1060 and group tree 1062,this embodiment illustrates a grouping of nodes A and E from transformtree 1060 into the same group (e.g., group tree 1062). (1) Determine ifA or E is already in a group. Here, E is in a group with B and F. (2)Determine A's group parent by finding a parent or ancestor of A thatbelongs in the same group as E. Here, A has no parents, so assign A tothe Group Root 1064 (see FIG. 10P). (3) Determine if A is a parent orancestor of any of the other nodes in the group. Here, A finds B as achild. Therefore, B sets its group parent to A (see FIG. 10Q).Additionally, A finds F as a descendant. A and F belong in the samegroup (e.g., group tree 1062). Therefore, F sets its group parent to A.(see FIG. 10R).

FIGS. 10S-10W illustrate an embodiment of adding grouped nodes toanother group. For the transform tree 1070 and group tree 1072 in FIG.10S, this embodiment illustrates a grouping of nodes C and G into thesame group. (1) Determine if C or G is already in a group. Here, C is ina group with D. G is in a group with B and F. (2) Since both C and G arein groups, merge the two into a single group by arbitrarily choosing thefinal group to be C and D's group (e.g., Group Root 1074). (3) Only addthe immediate children of the old Group Root 1076. This means add just Band F to C's group. (4) Add B to C's group. (4a) determine B's groupparent by finding a parent or ancestor of B that belongs in the samegroup as C. Here, there is no group parent. Therefore, add B as a childof Group Root 1074 (See FIG. 10T). (4b) Determine if B is a parent orancestor of any of the other nodes in the group. Here, B finds C as achild. B and C belong in the same group. C sets its group parent to B(See FIG. 10U). (5) Add F to C's group (NOTE: this is not arbitrary. Gwill never be chosen to be added next, it will just get pulled alongwith F). (5a) Determine F's group parent by finding a parent or ancestorof F that belongs in the same group as C. Here, F finds D as a parent. Fsets its group parent to D (See FIG. 10V). Note that G is automaticallyput into the same group. (5b) Determine if F is a parent or ancestor ofany of the other nodes in the group. Here, F finds G, but G's parent isalready F. (6) Delete the old Group Root 1076 (see FIG. 10W).

Transforming a Group

In some embodiments, any transform applied directly to a node thatbelongs in a group may transform the group along with it. Indirecttransforms done to a group node may not transform the group. Forexample, if the parent node is not in a group, its child is in a group,and the parent is transformed, then the nodes in the child's group DONOT transform with it even though the child's transform is changing. Ifmoving the whole group was desired, then the parent ought to be added tothe group.

To calculate the transform to apply to a group, the transform to usemust be applied to each immediate child of the Group Root.

FIG. 10X-10Y illustrates a sample calculation, according to someembodiments of the present disclosure. For the given transform tree 1080and group tree 1082 as depicted in FIG. 10X, this embodiment applies atransform to node A. Since A belongs in the group tree 1082, nodes C, D,and E are expected to be transformed with node A. Additionally, node B,which does not belong in the group, must also transform with the groupbecause it is a child of A. We are going to denote A's new world matrixas: Maw′, where Maw′ stands for “matrix a world new.” We are going todenote A's old world matrix as: Maw, where Maw stands for “matrix aworld old.” We are going to denote the matrix to apply to A to get Maw′as: Mgw, where Mgw stands for “matrix group world.” The goal is to finda matrix that we can set to the Group Root (Mgw) where when it's appliedto node A's world matrix, it will equal node A's new matrix. This samematrix is applied to the other immediate children of the Group Root.

FIG. 10Y illustrates a sample calculation for apply a transform to nodeA. From FIG. 10Y, we care about Node A because it is the one whosematrix changed. So:

Maw′=Mgw*Maw

We know Maw′ and Maw, solve for Mgw

Mgw=Maw′*Maw{circumflex over ( )}−1

So, for each child node of the group root, set the world transform ofthe node to: Child node new world matrix=Mgw*(child node old worldmatrix). For this example, apply Mgw to the two immediate children ofgroup Root, A and E:

$\begin{matrix}{{{Maw}’} = {{Mgw}*{Maw}}} \\{\left. {{{= \left( {Maw} \right.}’}*{{Maw}\hat{}{- 1}}} \right)*{Maw}} \\{{= {Maw}}’}\end{matrix}$ $\begin{matrix}{{{Mew}’} = {{Mgw}*{Mew}}} \\{\left. {{{{= \left( {Maw} \right.}’}*{Maw}} - 1} \right)*{Mew}}\end{matrix}$

Note that the final world transform of A is what we wanted, Maw′. Andsince transforms propagate down the transform tree, all children nodes(whether they're part of the same group or not) are updated. This meansnode B is also updated

FIG. 10Z-10AA illustrates a sample calculation, according to someembodiments of the present disclosure. For the given transform tree 1090and group tree 1092 as depicted in FIG. 10Z, this embodiment applies atransform to node C. Since C belongs in the group, nodes A, D, and E areexpected to be transformed with it. Additionally, node B, which does notbelong in the group, must also transform with the group because it is achild of A. We are going to denote C's new world matrix as: Mcw′, whereMcw′ stands for “matrix c world new.” We are going to denote C's oldworld matrix as: Mcw, where Mcw stands for “matrix c world old.” We aregoing to denote the matrix to apply to c to get Maw′ as: Mgw, where Mgwstands for “matrix group.” The goal is to find a matrix that we can setto the Group Root (Mgw) where when it's applied to node C's world matrixit will equal node C's new matrix. This same matrix is applied to theother immediate children of the Group Room.

In referring to FIG. 10AA, we care about Node C because it's the onewhose matrix changed. So Mcw′=Mgw*Mcw. We know Mcw′ and Mcw, solve forMgw. Mgw=Mcw′ *Mcw{circumflex over ( )}−1. So, for each child node ofthe group root, set the world transform of the node to: “Child node newworld matrix=Mgw*(child node old world matrix). For this example, recallwhat Mcw equals (Mcw=Ma1*Mb1*Mc1). Combine that withMgw=Mcw*Mcw{circumflex over ( )}−1 and we get Mgw=Mcw′*(Ma1*Mb1*Mc1){circumflex over ( )}−1 which Mgw=Mcw′ *Mc1{circumflexover ( )}−1*Mb1{circumflex over ( )}−1*Ma1{circumflex over ( )}−1. Anddo not forget that since And E are root nodes, Maw=Ma1 and Mew=Me1. Withthat, we can find the new transforms of every node. Apply Mgw to the twoimmediate children of the Group Root, E and A:

-   -   (E is a transform tree root node, so Mew==Me1).

$\begin{matrix}{{{Mew}’} = {{Mgw}*{Mew}}} \\{\left. {{{= \left( {Mcw} \right.}’}*{Mc}{1\hat{}{- 1}}*{Mb}{1\hat{}{- 1}}*{Ma}{1\hat{}{- 1}}} \right)*{Mew}} \\{\left. {{{= \left( {Mcw} \right.}’}*{Mc}{1\hat{}{- 1}}*{Mb}{1\hat{}{- 1}}*{Ma}{1\hat{}{- 1}}} \right)*{Me}1}\end{matrix}$

-   -   (A is a transform tree root node, so Maw==Ma1)

$\begin{matrix}{{{Maw}’} = {{Mgw}*{Maw}}} \\{\left. {{{= \left( {Mcw} \right.}’}*{Mc}{1\hat{}{- 1}}*{Mb}{1\hat{}{- 1}}*{Ma}{1\hat{}{- 1}}} \right)*{Maw}} \\{\left. {{{= \left( {Mcw} \right.}’}*{Mc}{1\hat{}{- 1}}*{Mb}{1\hat{}{- 1}}*{Ma}{1\hat{}{- 1}}} \right)*{Ma}1}\end{matrix}$

-   -   (Since A is a parent of other nodes, update the world transforms        of the child nodes B and C)

$\begin{matrix}{{{{{Mbw}’} = {Maw}}’}*{Mb}1} \\{\left. {{{= \left( {Mcw} \right.}’}*{Mc}{1\hat{}{- 1}}*{Mb}{1\hat{}{- 1}}} \right)*{Mb}1} \\{{{= {Mcw}}’}*{Mc}{1\hat{}{- 1}}}\end{matrix}$ $\begin{matrix}{{{{{Mcw}’} = {Mbw}}’}*{Mc}1} \\{\left. {{{= \left( {Mcw} \right.}’}*{Mc}{1\hat{}{- 1}}} \right)*{Mc}1} \\{{= {Mcw}}’}\end{matrix}$

-   -   Notice that the final world transform of node C is what we        wanted, Mcw′.

In some embodiments, the transform tree and transform groupfunctionality may be implemented within the Universe itself. In someembodiments, other applications (e.g., application(s) 140, icon gridapplication 260, status bar app 270, social panel app 280, store panelapp 290, etc.) may not need this kind of functionality since theUniverse may be managing this functionality. Other suitable methods orsystems utilizing group trees and/or transforming groups may be used.

Placed Launcher

In some embodiments, users may be able to place a launcher within theirlandscape where they want to launch an application or pin the launcherpersistently. This allows the launcher to be a natural extension of theuser's environment and creates a clear connection between the digitaland physical world. A placed launcher in its open state may have thesame layout and structure as a normal launcher except that it is tied toa location and may be scaled up and/or down. In some embodiments, when auser creates a placed launcher, the user has the option of pinning thelauncher persistently to that location. In this case, the launcher maystay in that location until the user moves or removes it. The normallauncher may still spawn when the user hits the home button even if alauncher has been pinned. In some embodiments, users are able to pinmultiples launchers (so they may, for example, put them in places theylaunch content the most). Pinned launchers may show the recentapplications that have been launched from that location. This allowseach launcher to be tailored to the space that it is in. In someembodiments, the behavior of pinning a launcher application to alocation is simulated by the extractable content manager (ECM) whichgives the appearance of pinning launcher icons. It does this byliterally copying the icon into a separate Prism and allowing the userto move that Prism around.

In some embodiments, since the launcher may be an application and thepanels of the launcher carousel and the launcher menus may be presentedinside separate Prisms, the Prisms themselves may be placed within theuser's landscape.

FIG. 11 shows an example view of a placed launcher, according to someembodiments. As shown in FIG. 11 , launcher menu 1100 may include anicon grid panel 1120, a minimized social panel 1110 and a minimizedstore panel 1130. The icon grid panel 1120 may actively display icons ofapplications a user may launch with the minimized social panel 1110 tothe left and the minimized store panel 1130 to the right of the icongrid panel 1120. An image icon 1140 may indicate to the user that thelauncher has been placed at the particular location, in this example, ontop of a stool. A different image icon 1140 may indicate a pinnedlauncher, as opposed to a placed launcher. In some embodiments,different numbers of panels, and/or different content for the panels maybe used.

In some embodiments, pinned launcher menus are persistently stored andmay remain pinned at the location until the user move or removes thelauncher from the location. Even if the user shuts down the mixedreality system, upon restart of the mixed reality system, a launchermenu may be displayed at the location after the system is fullylaunched, assuming a set of criteria is met, such as the user is in thevicinity of the pinned launcher menu. In some embodiments, applicationslaunched from a placed or pinned launcher, may spawn and display in thesame location as the pinned or placed launcher. In some embodiments,once the user selects a landscape application, the launcher maydisappear and the application (e.g., a Prism) may appear in the samelocation.

If the user chooses to pin the launcher persistently to the location,the launcher may remain there and the user may interact with thelauncher and launch applications from that location until the user movesor removes the launcher from the location. If the user does not interactwith the launcher, it may go into its idle minimize state. When the useris not interacting with the launcher, the state of the placed or pinnedlauncher may be changed to a minimized/idle state. From here the placedlauncher may show things like the time, featured store content, featuredsocial contacts, branded moments/experiences, and/or other content.

FIG. 12 shows example types of content that may be displayed while aplaced launcher is in an idle/sleeping state, according to someembodiments. A standard view of the placed launcher in itsminimized/sleeping state may display a time 1210 showing the currenttime and date. When the user hovers over the placed launcher with a headpose and/or clicks/selects the placed launcher, the launcher may changestate from sleeping to active and rendering and launch into its fullactive view, for example, showing at least an icon grid andcorresponding panels. In some embodiments, featured store content 1220may be displayed in place of the standard view time and date 1210.Featured store content 1220 may be a pre-downloaded preview of anapplication that gives a quick interactive glimpse of the product. Othertypes of content that may be displayed may include magical experiencessuch as, for example, a child's toy 1230 or a balloon with dailymessages 1240. These magical experiences may include a content andinspiration for users of the device. Artists, musicians and influencersmay create these content.

Secondary UI—Application Options Displayed in a Carousel

FIG. 13 shows an example Secondary UI volume, according to someembodiments. Applications options menu 1310 may show the differentinteractions a user may have with the content provided by an applicationthat is rendered into a Prism. The application options menu 1310 may bedisplayed within the volumetric space of the Secondary UI volume 220from FIG. 2 . In some embodiments, the application options menu may be acarousel with the center object always highlighted. When the user swipesto the next item, the icon may become more visible and display theappropriate text describing the option. The application title 1320 mayhelp the user to recognize the application that is providing thecontent. Here, as an example, the application is a 2D Internet browserdisplaying a web page. The settings 1330 may display the applicationsettings and/or link contextually to the universe settings application.The close application 1340 may close/remove the application from theuser's landscape. The share 1350 may allow users to start a sharingsession with this application to other users. The download 1360 maydownload content displayed within the Prism. For example, the user maywant to download an article displayed within the 2D Internet browser.One of ordinary skill in the art may appreciate that the applicationoptions menu 1310 may include more options or fewer options and could bein a different order than what has been disclosed and shown.

Body Dynamics

Body dynamics refer to how a displayed virtual content, anchoredrelative to a User's body may be managed and displayed to the user.

FIG. 14 shows an example of body dynamics, according to someembodiments. Body dynamics may refer to how the user application/userinterface/content 1480 moves in relationship to the user 1490. There maybe different types of body dynamics that may have different types ofuser interface effects. The body dynamic types may also be referred toas anchor types for how Prisms and/or applications within the Prisms areanchored and displayed in the user's landscape.

FIG. 15 shows different types of body dynamics, according to someembodiments. One example of a type of body dynamic is a world lock 1520,where there are no body dynamics. In some embodiments, generic Prismsare world locked and may not have a body dynamic unless the user choosesto activate a follow behavior from the application options. Anotherexample of a type of body dynamic is a Billboard 1530. In someembodiments, a billboard is when the angle and/or orientation of theapplication content changes to maintain a fixed orientation and/or anglerelative to the user. The relative angle and/or orientation may bedetermined any number of ways. For example, the virtual content may havea pre-defined forward or front direction. The system may require theforward vector of the virtual content to point at the user's forwarddirection, as determined by head pose. In some embodiments, thebillboard behavior maintains a fixed location in 3D space, whereas otherembodiments, may allow the location to vary. In some embodiments, abillboard dynamic may also include an edge billboard 1540, which onlybegins turning toward the user if the user would be viewing the contentfrom too oblique of an angle. For example, the application remainsstatic until the user is looking at a web page from a sideways anglethat makes the text difficult to read. The application follows to theedge of where the user pushes the frustum cone as projected from theplanar surface of the front of a Prism. Parameters for an edge billboard(e.g., application specific parameters) may include a frustum angle, afollowing behavior such as centers on user or moves to edge of whereuser pushes frustum cone, a force field (push) Boolean, a force fieldactivate distance, a speed, and/or a flip Boolean.

Body dynamic type of Follow 1550 is another type of body dynamic. Insome embodiments, follow 1550 is a specific mode that a single object(e.g., a Prism) may be put in by the user, for example, temporarilyattaching the object to the user's head position. This is useful when auser wants an object to move along with their head positions. A featureof follow is that it ignores the pitch (e.g., up/down rotation) of theuser's head, which gives the user greater freedom to look around attheir physical space. If the user looks to the left or right, the objectmay re-center in their field of view. In some embodiments, thisre-centering has a lerp value, wherein it takes time to move and feelssomewhat loosely attached to the user. In some embodiments, the Followbehavior may directly mimic the user's head movement, for example, on aone to one movement. In some embodiments, follow may be accessed andactivated by the application options menu on a Prism-by-Prism basis. Insome embodiments, the Follow mode may be a default assignment to a Prismand/or be a non-temporary property of the Prism (e.g. fixed as a FollowPrism until a user, for example, changes the Prism properly).

In some embodiments, the Follow 1550 body dynamic type may have one oftwo settings such as a lazy headlock 1560 and an external sensor 1570.During a lazy headlock 1560 setting, a single object, Prism, orapplication may be attached to the user's head position, similar to afollow 1550 setting, however, the pitch parameter may be unlocked by theuser so that if the user looks up and down, the object may follow aswell. During an external sensor 1570 setting, an object, Prism, orapplication may follow the user based on data from an external sensor.In some embodiments, an inertial measurement unit (IMU) may be includedin a belt pack unit of the user. In some embodiments, an IMU is a unitin an electronics module which collects angular velocity and linearacceleration data which is sent to a main processor. The IMU housing maycontain two separate sensors. The first sensor may be the accelerometertriad. It may generate three analog signals describing the accelerationsalong each of its axes produced by and acting on the user. The secondsensor may be an angular rate sensor triad. It may also output threeanalog signals. These signals may describe the user's angular rate abouteach of the sensor axes. This IMU would allow the mixed reality systemto have Prisms that are relative to the body instead of having to berelative to the user's head pose.

For the fade 1580 body dynamic setting, at a specified distance before aclipping plane, content may begin to dissolve/disappear. At thespecified distance before the clipping plane, in some embodiments, wherethe mixed reality system may no longer render content comfortably,and/or as the user gets near that cutoff point, the content should fadeout smoothly, rather than turn off abruptly. In some embodiments, fadingmay occur when the user is close to the clipping plane (e.g., exactmeasurement may depend upon hardware). Virtual content in the landscapemay have this effect when the user passes through or gets too close tothe content.

Simultaneously Launching and Placing an Application

In some embodiments, it may be beneficial to simultaneously launch andplace an application in a mixed reality environment. In one embodiment,every time a cursor moves over a web page, a browser engine may performa hit test on the web page content to see what is under the cursorposition. The hit test may detect elements in the web page and thenanalyzes what kind of element it is. Every html element may havedifferent parameters, some of which are common to all elements, some ofwhich are common to multiple elements and some of which areelement-specific. In the case of links, the most common and mostrecommended element is the use of an anchor tag <a href=“mylink”>. Thisanchor tag may be used with any element, image or text. At any time, wemay request the element type that the cursor is currently on top of. Thebrowser engine may cache the result of the hit test and pass the node ornode type to the browser application. In some embodiments, this cachingof the results of the hit test may be performed when a user selects alink using an extraction trigger.

In another embodiment, when a cursor is moved over any element in a webpage that is clickable, the web engine may call an API to the browserapplication to request that the cursor change from an “arrow” or“i-beam” to a “hand”. When this API is called, the browser applicationmay then request information about what kind of element the cursor isover.

FIG. 16 shows a flowchart for simultaneously launching and placing anapplication in a mixed reality environment, according to someembodiments. At 1610, a user may move a cursor over a link from a webpage displayed in a browser application displayed inside of a Prismwithin the user's landscape. The cursor may be moved by either ahandheld controller device such as a mouse, a totem, etc. or an eye gazewhere the user's eyes may be focused on the web page and sensors fromthe user's head-mounted system may detect that the user is looking overa link from the web page, or any other suitable method. The link may bea uniform resource locator (URL). The movement of the cursor over thelink and/or an eye gaze focusing on a web page (or any other suitableuser selection method) having a link may send a user input to anapplication rendering the content indicating that the user may have aninterest in content corresponding to the link.

At 1615, a web engine and/or a browser application may detect links thatthe user moves over with a cursor or an eye gaze. Upon detecting links,the web engine may notify a browser application that the user may beinterested in the content so that the browser application may requestthe Universe to create a mini display volume (e.g., a mini Prism). Themini Prism may be an initial default size of a standard 3D displayvolume managing unit (e.g., a Prism) for the purpose of serving as aplaceholder Prism for the browser application as the browser applicationis loading the content from the link into the mini Prism.

At 1620, the Universe creates the mini Prism and starts to load the miniPrism to show a page preview of the content being loaded. At 1625, theuser may select the link using an extraction trigger to indicate thatthe user is interested in viewing the content from which the link isdirected to. The use of the extraction trigger may indicate to theapplication that the user may be very interested in viewing the contentcorresponding to the link. The browser may cache the result of the hittest and pass the node or node type to the browser application to loadthe Prism. At 1630, the user may move the mini Prism off of the web pageusing the handheld controller. This movement of the mini Prism may senda user input to the Universe indicating that the user is moving the miniPrism.

At 1635, the mini Prism is in placement mode and the user may move themini Prism to place the mini Prism in a location in the user'slandscape. At 1640, the user may release the mouse button or extractiontrigger to place the mini Prism in a location in the user's landscape.The release of the mouse button or extraction trigger may send a userinput to the Universe indicating that the mini Prism is placed at alocation in the augmented reality environment (e.g., the user'slandscape).

At 1645, the mini Prism may be expanded to regular size (e.g., not themini Prism size) at the placement location with the page of the linkfully loaded within the Prism to be displayed to the user. In effect,the user is able to simultaneously open/launch and place an applicationin an augmented reality environment by selecting a link and placing amini Prism of the link in the mixed reality environment while in thebackground, the mixed reality system is loading and preparing the Prismfor display.

In an alternative embodiment, at 1670, after 1615, the user may selectthe link with an extraction trigger before the mini Prism is requestedby the application and created by the Universe at 1675. This may help toreduce system processing of creating a mini Prism before the useractually decide to extract the content from the link for display.

In an alternative embodiment, at 1680, after the user selects the linkwith the extraction trigger, the user may release the extraction triggerwithout moving the mini Prism. In this embodiment, at 1685, instead ofexpanding the mini Prism into the standard size Prism fully loaded withthe content from the link, the Universe may just create a new tab withinthe web browser application using the link, wherein the content of thelink has already been loaded and cached when the user selected the linkwith the extraction trigger equivalent user input.

What has been disclosed is a system and methods for managing anddisplaying virtual content in a mixed reality environment. Inparticular, the system and methods disclose a universe application thatmay function as a 3D windows manager by managing 3D windows, such asPrisms. Prisms are bounded volumes that are positioned in space (e.g. bythe user). Applications may render graphics into the Prism via theUniverse. The Universe renders the scene graphs, in cooperation withKali, and has full control over the placement, scaling, etc. of eachPrism. In some embodiments, the Universe may provide the ability toattach Prisms to walls and surfaces, register Prisms with the passableworld system, and/or control sharing of content between multiple usersof a mixed reality system. In some embodiments, the Universe alsomanages the Prisms themselves (e.g., creates Prisms, manages placementand/or snapping rules, provides basic user interface controls—closebutton, action bar, etc.), does not care what is inside the Prism,and/or keeps track of records of the Prism (e.g., what application ownsthe Prism, where to place the Prism, how the Prism is anchored—bodycentric, world fixed, etc.). In some embodiments, Prism behavior may bebased, in part, on anchors. In some embodiments, Prism behavior may bebased, in part, on placement (e.g. user placement through a userinteraction) and/or body dynamics (e.g. billboard, body centric, lazyheadlock, etc.).

The Universe may provide even more functionality as a 3D windows managerthan the standard 2D windows manager. One or more additional features(e.g. functionality) that the Universe may provide, in some embodiments,as a 3D windows manager (e.g. over a 2D windows manager) may include:

Persistence: An application sitting on a user's kitchen counter mayappear on the kitchen counter unless the user changes it. The user maynot have to re-launch the application every time the system is turnedon/off or every time the user leaves the room and comes back. Since theUniverse stores Prism information in the passable world system, theUniverse may restart the application sitting on the user's kitchen eachtime the user uses the mixed reality system and is in a close proximityto the application in the user's kitchen.

Application State relative to user:—An application may start,suspend/pause, re-start automatically—no explicit user action required.In contrast to a 2D windows manager, where a user interaction isrequired in order to change the operation state of an application (e.g.,user clicks the close button).

Locality—Real world vs. a 2D windows: A 2D windows manager may berestricted to a screen, regardless of where the screen may be placed.However, in a mixed reality environment, an application may be placed incontext to the real world, for example, relative to something else likea physical object in the user's environment, the user's body, a fixedlocation, etc.

Physicality: Prisms may move. Therefore, the movement and tracking ofthe movement of the Prisms needs to be managed (e.g., billboarding touser/body-centric, lazy billboarding, sway when move, collision bounce,etc.).

Private but interactable: 2D windows are private but they don'tinteract, or they interact but are not private. However, the Universemay enable both privacy and interaction of the 3D windows (e.g.,Prisms).

Additional Embodiments

Additional embodiments of the disclosure are described below. Theseadditional embodiments may incorporate elements from the embodimentsdisclosed above.

1. A method for starting a mixed reality system, the method comprising:

-   -   determining a current location of a user;    -   retrieving one or more prisms previously deployed at the current        location;    -   restoring the one or more prisms at the current location of the        user; and    -   displaying the one or more prisms restored at the current        location of the user.

2. The method of embodiment 1, wherein a prism is a volume of space thatvirtual content from an application is contained within.

3. The method of embodiment 2, wherein the application is an applicationinstance of the application when the application renders into more thanone prism.

4. The method of embodiment 1, wherein a prism represents a portion of ascene graph, the scene graph corresponding to the current location ofthe user.

5. The method of embodiment 4, wherein the scene graph comprises datafrom a first application and a second application

6. The method of embodiment 1, wherein retrieving the one or more prismspreviously deployed at the current location comprises:

-   -   retrieving instance data for the one or more prisms from an        external database; and    -   reconstructing a local prism database with the instance data for        the one or more prisms.

7. The method of embodiment 6, wherein the instance data for each prismincludes a data structure of prism properties defining the prism, theprism properties comprising at least one of a location, an orientation,an extent width, an extent height, an extent depth, a body dynamic, ananchor type, or an anchor position.

8. The method of embodiment 7, wherein the instance data for each prisminclude data of application specific properties comprising stateinformation of virtual content previously rendered into the prism by anapplication.

9. The method of embodiment 1, wherein restoring the one or more prismscomprises:

-   -   launching respective applications corresponding to the one or        more prisms previously deployed at the current location;    -   creating one or more new prisms corresponding to the one or more        prisms previously deployed; and    -   rendering respective virtual content into the one or more new        prisms.

10. The method of embodiment 9, further comprising displayingplaceholder prisms at locations corresponding to locations of the one ormore prisms previously deployed and displaying the one or more prismsfurther comprises replacing respective placeholder prisms with the oneor more new prisms having the respective virtual content.

11. The method of embodiment 1, further comprising:

-   -   updating the local prism database of the user with updated prism        instance data as the user interacts with the one or more prisms;        and    -   synchronizing the local prism database with the external        database.

12. A method for displaying virtual content in a 3D spatial environment,the method comprising:

-   -   receiving, from an application, a request to display a virtual        content in a 3D spatial environment;    -   creating a prism, wherein the prism is a volume of space        configured to bound the virtual content inside a boundary of the        prism;    -   receiving the virtual content from the application;    -   rendering the virtual content inside the boundaries of the        prism; and    -   associating the prism to an object in the 3D spatial        environment.

13. The method of embodiment 12, wherein boundaries of the prism are notdisplayed.

14. The method of embodiment 12, wherein the 3D spatial environment is aphysical real world environment of a user.

15. The method of embodiment 12, wherein the prism is createdautomatically having a set of functionalities.

16. The method of embodiment 15, wherein the set of functionalitiescomprises an association between the prism to the object in the 3Dspatial environment.

17. The method of embodiment 15, wherein the set of functionalitiescomprises one or more of a minimum size allowed for the prism, maximumsize allowed for the prism, and an aspect ratio for resizing the prism.

18. The method of embodiment 12, wherein the application renders a firstvirtual content into a first prism and a second virtual content into asecond prism, the first prism and the second prism being differentprisms.

19. The method of embodiment 18, wherein the prism does not overlap withother prisms in the 3D spatial environment.

20. The method of embodiment 12, wherein the prism is placed in contextto an object in the 3D spatial environment.

21. The method of embodiment 20, wherein the object is a user of anaugmented reality device.

22. The method of embodiment 12, wherein the prism comprises:

-   -   one or more universal features;    -   one or more application-specific features; and    -   wherein the one or more universal features and the one or more        application-specific features are selected from a list of        pre-approved options.

23. The method of embodiment 22, wherein the one or more universalfeatures ensure different applications interact in a consistent mannerwith one another.

24. A method for managing application states of virtual content in amixed reality system, the method comprising:

-   -   segmenting a 3D volume into a volumetric grid;    -   determining a first location of a user within the volumetric        grid;    -   determining a second location of an application within the 3D        volume;    -   calculating a distance between the user and the application        within the 3D volume; and    -   modifying a state of the application based at least in part on        the distance calculated between the user and the application.

25. The method of embodiment 24, wherein a radius of an active zonedefines a circular/spherical area around the user using a mixed realitydevice.

26. The method of embodiment 25, further comprising a buffer zone aroundan exterior of the active zone, wherein the buffer zone preventsintermittent or rapid changes to the state of the applications.

27. The method of embodiment 24, wherein modifying the state of theapplication is based at least in part on whether the application isoccluded by another application.

28. The method of embodiment 24, wherein modifying the state of theapplication is based at least in part on a head pose of the user.

29. The method of embodiment 24, wherein the distance of known positionsof applications within the cell is determined only for the cell the userusing the mixed reality device is in and the neighboring cells.

30. A method for opening and placing an application in an augmentedreality environment, the method comprising:

-   -   receiving a first user input from a user indicating a request        for new content;    -   launching an application to generate the content;    -   creating a mini display volume of a 3D display volume managing        unit, wherein a page preview is displayed in the mini display        volume, wherein the 3D display volume managing unit is created        simultaneously with the launching of the application;    -   receiving a second user input indicating a movement of the mini        display volume;    -   receiving a third user input indicating a placement of the mini        display volume at a location in the augmented reality        environment; and    -   expanding the 3D display volume managing unit in place of the        mini display volume at the location, the 3D display volume        managing unit displaying the content fully loaded within the 3D        display volume managing unit.

31. The method of embodiment 30, wherein the first user input is acursor movement over a link on a web page.

32. The method of embodiment 31, wherein the second user input is aselection of the link, and movement of the mini display volume.

33. The method of embodiment 30, wherein the 3D display volume managingunit replaces the mini display when the 3D display volume managing unitexpands in place of the mini display.

34. The method of embodiment 30, wherein the content is loaded into the3D display volume managing unit while the user is moving the minidisplay.

35. The method of embodiment 30, wherein the location is fixed to anobject in the augmented reality environment.

36. The method of embodiment 35, wherein the object is the user.

37. A method for managing virtual content, the method comprising:

-   -   receiving content from a content generating application;    -   displaying the content in a 3D spatial environment by a universe        application; and    -   constantly managing the display of the content in the 3D spatial        environment by the universe application.

38. A system, method, and computer program product for managing anddisplaying virtual content in a mixed reality system according to any ofthe inventive concepts disclosed herein.

39. The system, method, and computer program product of embodiment 38,further comprising displaying virtual content within a sub-set ofavailable 3D displayable space by displaying the virtual content withina volumetric display space, wherein boundaries of the volumetric displayspace are not displayed.

40. The system, method, and computer program product of embodiment 38,further comprising assigning to a Prism universal features andapplication selected features from a list of pre-approved options forconfigurations of display customizations by an application.

41. The system, method, and computer program product of embodiment 38,further comprising displaying virtual content into one or more Prisms,wherein the one or more Prisms do not overlap with one another.

42. The system, method, and computer program product of embodiment 38,further comprising changing a state of a Prism based at least in part ona relative position and location of the Prism to a user.

43. The system, method, and computer program product of embodiment 38,further comprising managing content creation in an application andmanaging content display in a separate application.

44. The system, method, and computer program product of embodiment 38,further comprising while placing a Prism in a mixed reality environment,opening an application that will provide content into the Prism.

45. The system, method, and computer program product of embodiment 38,further comprising assigning location, orientation, and extent data to aPrism for displaying virtual content within the Prism, the virtualcontent is 3D virtual content.

46. The system, method, and computer program product of embodiment 45,wherein the location is a coordinate of an anchor position of the Prismin a mixed reality environment.

47. The system, method, and computer program product of embodiment 45,wherein the orientation defines how the Prism is rotated relative to anobject, wherein the object is a wall.

48. The system, method, and computer program product of embodiment 45,wherein the extent data defines a size of the Prism.

49. The system, method, and computer program product of embodiment 38,further comprising pinning a launcher application to a real world objectwithin a mixed reality environment.

50. The system, method, and computer program product of embodiment 49,wherein the pinned launcher application launches content of theapplication within a Prism in a same location as the pinned launcherapplication.

51. The system, method, and computer program product of embodiment 38,further comprising assigning behavior type to each Prism, the behaviortype comprising at least one of a world lock, a billboard, an edgebillboard, a follow headlock, a follow based on external sensor, or afade.

52. The system, method, and computer program product of embodiment 38,further comprising identifying a most used content specific to a placedlocation of a launcher application.

53. The system, method, and computer program product of embodiment 38,further comprising displaying favorite applications, by a placedlauncher application, the favorite applications based at least in parton context relative to a location of the placed launcher.

54. A method comprising:

-   -   accessing a scene graph for a scene, wherein the scene graph        comprises one or more transform trees, each tree comprising a        plurality of nodes;    -   adding a tag to one or more nodes from a plurality of nodes        within the one or more transform trees, wherein the one or more        nodes tagged form a transform group, wherein the one or more        nodes tagged comprise a first tagged node and a second tagged        node; and    -   moving the first tagged node, wherein moving the first tagged        node causes the second tagged node to move.

55. A method of embodiment 54, wherein the second tagged node is not adirect descendant of the first tagged node.

56. A method of embodiment 54, wherein the first tagged node and secondtagged node are not from a same transform tree.

57. A method of embodiment 54, wherein the plurality of nodes comprisesa first node and a second node, the method further comprising parentingthe first node to the second node, wherein parenting the first node tothe second node does not cause the parented node to move.

58. A method of displaying virtual content in a 3D shared space, themethod comprising:

-   -   generating, by a first application, virtual content in the 3D        shared space; and    -   displaying, by a second application, the virtual content        generated by the first application, the first application and        the second application being different applications.

System Architecture Overview

FIG. 17 is a block diagram of an illustrative computing system 1400suitable for implementing one or more of the embodiments of the presentdisclosure. The computing system 1400 includes a bus 1406 or othercommunication mechanism for communicating information, whichinterconnects subsystems and devices, such as a processor 1407, a mainmemory 1408 (e.g., RAM), a static storage device 1409 (e.g., ROM), adisk drive 1410 (e.g., magnetic or optical), a communications interface1414 (e.g., modem or Ethernet card), a display 1411 (e.g., CRT or LCD),an input device 1412 (e.g., keyboard), and cursor control.

According to some embodiments, the computing system 1400 performsspecific operations by the processor 1407 executing one or moresequences of one or more instructions contained in the main memory 1408.Such instructions may be read into the main memory 1408 from anothercomputer readable/usable medium, such as the static storage device 1409or the disk drive 1410. In alternative embodiments, hard-wired circuitrymay be used in place of or in combination with software instructions toimplement the disclosure. Thus, embodiments are not limited to anyspecific combination of hardware circuitry and/or software. In oneembodiment, the term “logic” shall mean any combination of software orhardware that is used to implement all or part of the disclosure.

The term “computer readable medium” or “computer usable medium” as usedherein refers to any medium that participates in providing instructionsto the processor 1407 for execution. Such a medium may take many forms,including but not limited to, non-volatile media and volatile media.Non-volatile media includes, for example, optical or magnetic disks,such as the disk drive 1410. Volatile media includes dynamic memory,such as the main memory 1408. In some embodiments, the system may use asolid state drive (SSD) memory.

Common forms of computer readable media include, for example, floppydisk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EPROM,FLASH-EPROM, any other memory chip or cartridge, or any other mediumfrom which a computer can read.

In one embodiment, execution of the sequences of instructions topractice the disclosure is performed by a single computing system 1400.According to other embodiments, two or more computing systems 1400coupled by a communications link 1415 (e.g., LAN, PTSN, or wirelessnetwork) may perform the sequence of instructions required to practicethe disclosure in coordination with one another.

The computing system 1400 may transmit and receive messages, data, andinstructions, including program, e.g., application code, through thecommunications link 1415 via the communications interface 1414. Receivedprogram code may be executed by the processor 1407 as it is received,and/or stored in the disk drive 1410, or other non-volatile storage forlater execution. The computing system 1400 may communicate through adata interface 1433 to a database 1432 on an external storage device1431.

In the foregoing specification, the disclosure has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the disclosure. Forexample, the above-described process flows are described with referenceto a particular ordering of process actions. However, the ordering ofmany of the described process actions may be changed without affectingthe scope or operation of the disclosure. The specification and drawingsare, accordingly, to be regarded in an illustrative rather thanrestrictive sense.

What is claimed is:
 1. A method for managing application states ofvirtual content in a mixed reality system, the method comprising:segmenting a 3D volume into a volumetric grid; determining a firstlocation of a user within the volumetric grid; determining a secondlocation of an application within the 3D volume; calculating a distancebetween the user and the application within the 3D volume; and modifyinga state of the application based at least in part on the distancecalculated between the user and the application.
 2. The method of claim1, wherein a radius of an active zone defines a circular/spherical areaaround the user using a mixed reality device.
 3. The method of claim 2,further comprising a buffer zone around an exterior of the active zone,wherein the buffer zone prevents intermittent or rapid changes to thestate of the applications.
 4. The method of claim 1, wherein modifyingthe state of the application is based at least in part on whether theapplication is occluded by another application.
 5. The method of claim1, wherein modifying the state of the application is based at least inpart on a head pose of the user.
 6. The method of claim 1, wherein thedistance of known positions of applications within the cell isdetermined only for the cell the user using the mixed reality device isin and the neighboring cells.
 7. A method for opening and placing anapplication in an augmented reality environment, the method comprising:receiving a first user input from a user indicating a request for newcontent; launching an application to generate the content; creating amini display volume of a 3D display volume managing unit, wherein a pagepreview is displayed in the mini display volume, wherein the 3D displayvolume managing unit is created simultaneously with the launching of theapplication; receiving a second user input indicating a movement of themini display volume; receiving a third user input indicating a placementof the mini display volume at a location in the augmented realityenvironment; and expanding the 3D display volume managing unit in placeof the mini display volume at the location, the 3D display volumemanaging unit displaying the content fully loaded within the 3D displayvolume managing unit.
 8. The method of claim 7, wherein the first userinput is a cursor movement over a link on a web page.
 9. The method ofclaim 8, wherein the second user input is a selection of the link, andmovement of the mini display volume.
 10. The method of claim 7, whereinthe 3D display volume managing unit replaces the mini display when the3D display volume managing unit expands in place of the mini display.11. The method of claim 7, wherein the content is loaded into the 3Ddisplay volume managing unit while the user is moving the mini display.12. The method of claim 7, wherein the location is fixed to an object inthe augmented reality environment.
 13. The method of claim 12, whereinthe object is the user.
 14. A method comprising: accessing a scene graphfor a scene, wherein the scene graph comprises one or more transformtrees, each tree comprising a plurality of nodes; adding a tag to one ormore nodes from a plurality of nodes within the one or more transformtrees, wherein the one or more nodes tagged form a transform group,wherein the one or more nodes tagged comprise a first tagged node and asecond tagged node; and moving the first tagged node, wherein moving thefirst tagged node causes the second tagged node to move.
 15. The methodof claim 14, wherein the second tagged node is not a direct descendantof the first tagged node.
 16. The method of claim 14, wherein the firsttagged node and second tagged node are not from a same transform tree.17. The method of claim 14, wherein the plurality of nodes comprises afirst node and a second node, the method further comprising parentingthe first node to the second node, wherein parenting the first node tothe second node does not cause the parented node to move.